File generation apparatus and file generation method as well as reproduction apparatus and reproduction method

ABSTRACT

There is provided a file generation apparatus and a file generation method as well as a reproduction apparatus and a reproduction method by which a file for efficiently storing quality information of a depth-related image at least including a depth image can be generated. A segment file generation unit generates a file in which quality information representative of quality of a depth-related image including at least a depth image is disposed in a form divided for each kind. The present disclosure can be applied to a file generation apparatus of an information processing system or the like by which a segment file and an MPD file of a video content are distributed, for example, in a method that complies with MPEG-DASH.

TECHNICAL FIELD

The present disclosure relates to a file generation apparatus and a filegeneration method as well as a reproduction apparatus and a reproductionmethod, and particularly to a file generation apparatus and a filegeneration method as well as a reproduction apparatus and a reproductionmethod by which a file for efficiently storing quality information of adepth-related image at least including a depth image can be generated.

BACKGROUND ART

As a technique for implementing stereoscopic vision, a technology thatuses a texture image and a depth image is available. The depth image isan image in which a value for representing a position of each pixel in adepthwise direction of an image pickup object is a pixel value.

In such a technology as just described, there is a case in which, inorder to implement natural stereoscopic vision, an occlusion image isused as additional information. The occlusion image is a texture imagein an occlusion region that is a region of an image pickup object thatdoes not exist in the texture image, namely, of an image pickup objectthat is invisible from a view point of the texture image (for example,an image pickup object hidden by a nearer image pickup object). By usingnot only a texture image and a depth image but also an occlusion image,a 3D image that implements stereoscopic vision in the case where peepingin from points of view different from each other or the like isperformed can be generated.

The texture image and the depth image can be transmitted, for example,by an existing MPEG-DASH (Moving Picture Experts Group Dynamic AdaptiveStreaming over HTTP standard) method (for example, refer to NPL 1).

In this case, a DASH client selects and acquires a depth image having amaximum acceptable bit rate from depth images of a plurality of bitrates stored in a DASH server taking a transmission path and a bufferamount of the DASH client itself into consideration.

However, in the case where the influence of the bit rate of a depthimage on the picture quality of a 3D image is less, in the case wherethe variation amount of a pixel value of a depth image is small or in alike case, the picture quality of a 3D image is not varied very much bythe bit rate of a depth image. Accordingly, in this case, if a DASHclient selects and acquires a depth image having a maximum acceptablebit rate, then a transmission path and a buffer become useless.

On the other hand, ISO/IEC 23001-10 proposes to store qualityinformation representative of one or more kinds of quality of a textureimage into an MP4 file of the ISO base media file format.

CITATION LIST Non Patent Literature [NPL 1]

-   ISO/IEC 23009-1 Dynamic adaptive streaming over HTTP (DASH) Part 1:    Media presentation description and segment formats, April 2012

SUMMARY Technical Problem

As described above, also in regard to a depth-related image at leastincluding a depth image, it is demanded that quality information isstored into a file similarly to a texture image such that a DASH clientacquires a depth-related image having an appropriate bit rate using thequality information. However, it is not considered to store qualityinformation of a depth-related image efficiently into a file.

The present disclosure has been made in view of such a situation asdescribed above and makes it possible to generate a file into whichquality information of a depth-related image at least including a depthimage is stored efficiently.

Solution to Problem

A file generation apparatus of a first aspect of the present disclosureis a file generation apparatus including a file generation unitconfigured to generate a file in which quality informationrepresentative of quality of a depth-related image including at least adepth image is disposed in a form divided for each kind.

A file generation method of the first aspect of the present disclosurecorresponds to the file generation apparatus of the first aspect of thepresent disclosure.

In the first aspect of the present disclosure, a file in which qualityinformation representative of quality of a depth-related image includingat least a depth image is disposed in a form divided for each kind isgenerated.

A reproduction apparatus of a second aspect of the present disclosure isa reproduction apparatus including an acquisition unit configured toacquire, from a file in which quality information representative ofquality of a depth-related image including at least a depth image isdisposed in a form divided for each kind, the quality information of agiven kind.

A reproduction method of the second aspect of the present disclosurecorresponds to the reproduction apparatus of the second aspect of thepresent disclosure.

In the second aspect of the present disclosure, from a file in whichquality information representative of quality of a depth-related imageincluding at least a depth image is disposed in a form divided for eachkind, the quality information of a given kind is acquired.

It is to be noted that the file generation apparatus of the first aspectand the reproduction apparatus of the second aspect of the presentdisclosure can be implemented by causing a computer to execute aprogram.

Further, in order to implement the file generation apparatus of thefirst aspect and the reproduction apparatus of the second aspect of thepresent disclosure, the program for being executed by the computer canbe provided by transmitting the program through a transmission medium orby recording the program on a recording medium.

Advantageous Effects of Invention

With the first aspect of the present disclosure, a file can begenerated. Further, with the first aspect of the present disclosure, afile into which quality information of a depth-related image at leastincluding a depth image is stored efficiently can be generated.

With the second aspect of the present disclosure, quality informationcan be acquired from a file in which quality information of adepth-related image at least including a depth image is storedefficiently.

It is to be noted that the effects described here are not necessarilyrestrictive and the effects may be any one of the effects described inthe present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view illustrating an overview of an information processingsystem according to a first embodiment to which the present disclosureis applied.

FIG. 2 is a view illustrating an occlusion image.

FIG. 3 is a view illustrating a hierarchical structure of an MPD file.

FIG. 4 is a block diagram depicting an example of a configuration of afile generation apparatus of FIG. 1.

FIG. 5 is a view depicting an example of a segment file in the firstembodiment.

FIG. 6 is a view depicting an example of a description ofQualityMetricsSampleEntry in the first embodiment.

FIG. 7 is a view illustrating description contents ofQualityMetricsSampleEntry of FIG. 6.

FIG. 8 is a view depicting an example of metric_code.

FIG. 9 is a view depicting an example of a representation element of theMPD file in the first embodiment.

FIG. 10 is a view depicting an example of description of the MPD file inthe first embodiment.

FIG. 11 is a view illustrating a leva box in the first embodiment.

FIG. 12 is a flow chart illustrating a file generation process.

FIG. 13 is a block diagram depicting an example of a configuration of astreaming reproduction unit.

FIG. 14 is a flow chart illustrating a first example of a reproductionprocess in the first embodiment.

FIG. 15 is a flow chart illustrating a second example of thereproduction process in the first embodiment.

FIG. 16 is a view depicting an example of a segment file in a secondembodiment of the information processing system to which the presentdisclosure is applied.

FIG. 17 is a view depicting an example of a segment file in a thirdembodiment of the information processing system to which the presentdisclosure is applied.

FIG. 18 is a view depicting an example of description of an MPD file inthe third embodiment.

FIG. 19 is a flow chart illustrating a first example of a reproductionprocess in the third embodiment.

FIG. 20 is a flow chart illustrating a second example of thereproduction process in the third embodiment.

FIG. 21 is a view depicting an example of a segment file in a fourthembodiment of the information processing system to which the presentdisclosure is applied.

FIG. 22 is a view depicting an example of a segment file in a fifthembodiment of the information processing system to which the presentdisclosure is applied.

FIG. 23 is a view depicting an example of a segment file in a sixthembodiment of the information processing system to which the presentdisclosure is applied.

FIG. 24 is a view depicting an example of a description of an MPD filein the sixth embodiment.

FIG. 25 is a view depicting an example of a segment file in a seventhembodiment of the information processing system to which the presentdisclosure is applied.

FIG. 26 is a view depicting an example of a configuration of a sample ofa track of FIG. 25.

FIG. 27 is a view depicting an example of a configuration of a moov boxof a depth file.

FIG. 28 is a view depicting an example of a description ofQualityMetricsConfigurationBox.

FIG. 29 is a view depicting an example of a description ofQualityMetricsSampleEntry in description in QualityMetricsSampleEntry inthe seventh embodiment.

FIG. 30 is a view depicting an example of a description ofSubsampleInformationBox.

FIG. 31 is a view depicting an example of a description ofSubsampleReferenceBox.

FIG. 32 is a view depicting an example of a description of an MPD filein the seventh embodiment.

FIG. 33 is a view depicting an example of a segment file in an eighthembodiment of the information processing system to which the presentdisclosure is applied.

FIG. 34 is a view depicting a first example of a description of a levabox.

FIG. 35 is a view illustrating a first example of a level and asubsample associated with each other by the leva box of FIG. 34.

FIG. 36 is a view depicting a second example of a description of theleva box.

FIG. 37 is a view illustrating a second example of a level and asubsample associated with each other by the leva box of FIG. 36.

FIG. 38 is a view depicting an example of a description of a subsamplegroup entry.

FIG. 39 is a view illustrating a third example of a level and asubsample associated with each other by the leva box.

FIG. 40 is a block diagram depicting an example of a configuration ofhardware of a computer.

DESCRIPTION OF EMBODIMENTS

In the following, modes for carrying out the present disclosure(hereinafter referred to as embodiments) are described. It is to benoted that the description is given in accordance with the followingorder.

1. First Embodiment: Information Processing System (FIGS. 1 to 15)

2. Second Embodiment: Information Processing System (FIG. 16)

3. Third Embodiment: Information Processing System (FIGS. 17 to 20)

4. Fourth Embodiment: Information Processing System (FIG. 21)

5. Fifth Embodiment: Information Processing System (FIG. 22)

6. Sixth Embodiment: Information Processing System (FIGS. 23 and 24)

7. Seventh Embodiment: Information Processing System (FIGS. 25 to 32)

8. Eighth Embodiment: Information Processing System (FIGS. 33 and 39)

9. Ninth Embodiment: Computer (FIG. 40)

First Embodiment (Overview of Information Processing System)

FIG. 1 is a view illustrating an overview of an information processingsystem according to a first embodiment to which the present disclosureis applied.

An information processing system 10 of FIG. 1 is configured byconnecting a Web server 12 as a DASH server connected to a filegeneration apparatus 11 and a video reproduction terminal 14 as a DASHclient through the Internet 13.

In the information processing system 10, the Web server 12 distributes afile of a video content generated by the file generation apparatus 11 tothe video reproduction terminal 14 by a method that complies withMPEG-DASH.

In particular, the file generation apparatus 11 encodes image data andsound data of a texture image, a depth image, and an occlusion image ofa video content, metadata including quality information of the depthimage and the occlusion image and so forth with one or more bit rates.

It is assumed that, in the present specification, two bit rates of 8Mbps and 4 Mbps are available for the texture image; two bit rates of 2Mbps and 1 Mbps are available for the depth image; and one bit rate of 1Mbps is available for the occlusion image. Further, in the followingdescription, in the case where the depth image and the occlusion imageneed not be distinguished from each other specifically, the images arereferred to as depth occlusion images.

The file generation apparatus 11 files an encoded stream of image dataand sound data of the respective bit rates generated as a result ofencoding in a unit of time of several seconds to approximately tenseconds called segment in an ISO base media file format. The filegeneration apparatus 11 uploads a segment file that is an MP4 file ofimage data and sound data generated as a result of the process describedabove to the Web server 12.

Further, the file generation apparatus 11 divides an encoded stream ofmetadata including quality information of a depth occlusion image in aunit of a segment for each kind of depth occlusion images and files thedivisions of the encoded stream in the ISO base media file format. Thefile generation apparatus 11 uploads the segment files of the metadatagenerated as a result of the process just described to the Web server12.

Further, the file generation apparatus 11 generates an MPD (MediaPresentation Description) file (management file) for managing a segmentfile group of a video content. The file generation apparatus 11 uploadsthe MPD file to the Web server 12.

The Web server 12 stores the segment files and the MPD files uploadedfrom the file generation apparatus 11 therein. The Web server 12transmits a stored segment file or MPD file to the video reproductionterminal 14 in response to a request from the video reproductionterminal 14.

The video reproduction terminal 14 (reproduction apparatus) executescontrolling software for streaming data (hereinafter referred to ascontrolling software) 21, video reproduction software 22, clientsoftware for HTTP (Hypertext Transfer Protocol) access (hereinafterreferred to as access software) 23 and so forth.

The controlling software 21 is software for controlling data to bestreamed from the Web server 12. In particular, the controlling software21 causes the video reproduction terminal 14 to acquire an MPD file fromthe Web server 12.

Further, the controlling software 21 issues a transmission request foran encoded stream of a segment file of a reproduction target to theaccess software 23 on the basis of time of a reproduction targetdesignated by the video reproduction software 22 and reproduction targetinformation representative of a bit rate or the like.

The video reproduction software 22 is software for reproducing anencoded stream acquired from the Web server 12. In particular, the videoreproduction software 22 designates reproduction target information inwhich an encoded stream of metadata is a reproduction target to thecontrolling software 21. Then, when a notification of reception startingof the encoded stream of metadata is received from the access software23, the video reproduction software 22 decodes the encoded stream ofmetadata received by the video reproduction terminal 14.

The video reproduction software 22 designates reproduction targetinformation in which an encoded stream of image data or sound data of apredetermined bit rate is a reproduction target to the controllingsoftware 21 on the basis of quality information included in metadataobtained as a result of the decoding, a network bandwidth of theInternet 13 or the like. Then, when a notification of reception startingof the encoded stream of image data or sound data is received from theaccess software 23, the video reproduction software 22 decodes theencoded stream of image data or sound data received by the videoreproduction terminal 14.

The video reproduction software 22 outputs the image data of a textureimage obtained as a result of the decoding as it is. Further, the videoreproduction software 22 generates and outputs image data of a 3D imageusing the texture image and the depth image. Further, the videoreproduction software 22 generates and outputs image data of a 3D imageusing the texture image, the depth image, and the occlusion image.Further, the video reproduction software 22 outputs sound data obtainedas a result of the decoding.

The access software 23 is software for controlling communication withthe Web server 12 through the Internet 13 using HTTP. In particular, inresponse to an instruction of the controlling software 21, the accesssoftware 23 causes the video reproduction terminal 14 to transmit atransmission request for an encoded stream of a segment file of areproduction target. Further, the access software 23 causes the videoreproduction terminal 14 to start reception of an encoded streamtransmitted from the Web server 12 in response to the transmissionrequest, and supplies a notification of reception starting to the videoreproduction software 22.

It is to be noted that, since the present disclosure is an inventionrelating to image data and metadata of a video content, in thefollowing, description of storage and reproduction of a segment file ofsound data is omitted.

(Description of Occlusion Image)

FIG. 2 is a view illustrating an occlusion image.

If an image of a circular cylinder 41 and a cube 42 on the upper stageof FIG. 2 is picked up from a front direction indicated by an arrow mark51, then a texture image 61 on the left side on the lower stage of FIG.2 is obtained.

In the case where a texture image 62 picked up from a direction lookingin from the left indicated by an arrow mark 52 is to be generated usingthe texture image 61 and a depth image of the texture image 61, pixelvalues of the texture image 61 corresponding to pixels of the textureimage 62 are acquired on the basis of the depth image of the textureimage 61. Then, the texture image 62 is generated by determining thepixel values of the pixels of the texture image 62 as the pixel valuesof the texture image 61 corresponding to the pixels.

However, as indicated on the right side on the lower stage of FIG. 2, acorresponding texture image 61 does not exist in the texture image 62.In particular, an occlusion region 43 is generated which is a region ofan image pickup object (in the example of FIG. 2, a side face of thecube 42) that is not picked up upon image pickup from a directionindicated by an arrow mark 51 but is picked up upon image pickup from adirection indicated by an arrow mark 52. A texture image of theocclusion region 43 is an occlusion image.

Accordingly, the texture image 62 from a viewpoint different from thatof the texture image 61 can be generated by using the texture image 61,depth image of the texture image 61, and texture image of the occlusionregion 43. Further, a depth image of the texture image 62 can begenerated from the texture image 62 and the depth image of the textureimage 61. Therefore, a 3D image from a viewpoint different from that ofthe texture image 61 can be generated from the texture image 62 and thedepth image of the texture image 62.

(Description of MPD File)

FIG. 3 is a view illustrating a hierarchical structure of an MPD file.

In the MPD file, information of an encoding method and a bit rate of avideo content, a size of an image, a language of speech and so forth isdescribed in a hierarchical relationship in the XML format.

In particular, as depicted in FIG. 3, elements such as a period(Period), an adaptation set (AdaptationSet), a representation(Representation), segment information (SegmentInfo) and so forth areincluded hierarchically in the MPD file.

In the MPD file, a video content managed by the MPD file itself isdivided by a predetermined time range (for example, a unit such as aprogram, a CM (Commercial Message) or the like). The period element isdescribed for each of the divisions of the divided video content. Theperiod element has information of reproduction starting time of aprogram of the video content (data of a set of image data or sound datasynchronized with each other or the like), a URL (Uniform ResourceLocator) of the Web server 12 into which a segment file of the videocontent is to be stored and so forth.

The adaptation set element is included in the period element and groupsa representation element corresponding to a segment file group of thesame encoded stream of the video content corresponding to the periodelement. The representation element is grouped, for example, by a kindof data of the corresponding encoded stream. The adaptation set elementhas a use as a media type, a language, a subtitle, a dubbing and soforth common to the group.

The representation element is included in the adaptation set element forgrouping the representation element and is described for each segmentfile group of the same encoded stream of the video content correspondingto the period element in the upper hierarchy. The representation elementhas a bit rate, a size of an image and so forth common to the segmentfile group.

The segment information element is included in the representationelement and has information relating to respective segment files of thesegment file group corresponding to the representation.

(Example of Configuration of File Generation Apparatus)

FIG. 4 is a block diagram depicting an example of a configuration of thefile generation apparatus of FIG. 1.

The file generation apparatus 11 of FIG. 4 is configured from anacquisition unit 81, an encoding unit 82, a segment file generation unit83, an MPD file generation unit 84, and an upload unit 85.

The acquisition unit 81 of the file generation apparatus 11 acquires andsupplies image data of a texture image, a depth image, and an occlusionimage of a video content to the encoding unit 82. Further, theacquisition unit 81 acquires and supplies metadata including qualityinformation of encoded streams of depth images of 2 Mbps and 1 Mbps andan occlusion image of 1 Mbps to the encoding unit 82.

The encoding unit 82 encodes the image data of the texture imagesupplied from the acquisition unit 81 in 8 Mbps and 4 Mbps and encodesthe image data of the depth image in 2 Mbps and 1 Mbps. Further, theencoding unit 82 encodes the image data of the occlusion image in 1Mbps. Furthermore, the encoding unit 82 encodes metadata of the depthimages of 2 Mbps and 1 Mbps and the occlusion image of 1 Mbpsindividually at predetermined bit rates. The encoding unit 82 suppliesthe encoded streams generated as a result of the encoding to the segmentfile generation unit 83.

The segment file generation unit 83 files encoded streams of the textureimages, depth images, and occlusion image supplied from the encodingunit 82 in a unit of a segment for each bit rate to generate a segmentfile of image data.

Further, the segment file generation unit 83 (file generation unit)divides the encoded stream of metadata supplied from the encoding unit82 into two for each kind of depth occlusion image. Then, the segmentfile generation unit 83 disposes the divisions of the encoded stream ofmetadata into a different segment file in a unit of a segment togenerate a segment file of metadata.

In particular, the segment file generation unit 83 divides the encodedstream of metadata supplied from the encoding unit 82 into an encodedstream of metadata in a unit of a segment of the depth images of 2 Mbpsand 1 Mbps and another encoded stream of metadata in a unit of a segmentof the occlusion image of 1 Mbps. Then, the segment file generation unit83 individually files the encoded stream of metadata of a unit of asegment of the depth images of 2 Mbps and 1 Mbps and the encoded streamof metadata of a unit of a segment of the occlusion image of 1 Mbps togenerate a segment file of metadata. The segment file generation unit 83supplies the generated segment file to the upload unit 85.

The MPD file generation unit 84 (file generation unit) generates andsupplies an MPD file to the upload unit 85.

The upload unit 85 uploads the segment file supplied from the segmentfile generation unit 83 and the MPD file supplied from the MPD filegeneration unit 84 to the Web server 12.

(Example of Segment File)

FIG. 5 is a view depicting an example of a segment file generated by thesegment file generation unit 83 of FIG. 4.

As depicted in FIG. 5, the segment file generation unit 83 generates asegment file of a texture image of 8 Mbps as a texture file (texture1file) and generates a segment file of a texture image of 4 Mbps asanother texture file (texture2 file). Further, the segment filegeneration unit 83 generates a segment file of a depth image of 2 Mbpsas a depth file (depth1 file) and generates a segment file of a depthimage of 1 Mbps as a depth file (depth2 file). Further, the segment filegeneration unit 83 generates a segment file of an occlusion image of 1Mbps as an occlusion file (occlusion1 file).

Further, the segment file generation unit 83 generates a segment file ofmetadata including quality information of the depth images of 2 Mbps and1 Mbps as a quality file (quality1 file). In the quality file (quality1file), metadata including quality information of the depth image of 2Mbps and metadata including quality information of the depth image of 1Mbps are disposed in tracks different from each other (qualitytrack(depth1) and quality track(depth2)).

Furthermore, the segment file generation unit 83 generates a segmentfile of metadata including quality information of the occlusion image of1 Mbps as a quality file (quality2 file).

As described above, the segment file generation unit 83 files theencoded stream of metadata including quality information separately fordifferent kinds of depth occlusion images. Accordingly, in the casewhere the video reproduction terminal 14 generates a 3D image using thetexture images and the depth images, desired quality information of thedepth images can be acquired readily from the quality file (quality1file) of the depth image.

In contrast, in the case where encoded streams of quality information ofall depth occlusion images are filed collectively, quality informationof desired depth images is acquired from a file that includes alsoquality information of unnecessary occlusion images, and the acquisitionefficiency is low.

Further, in the case where encoded streams of quality information of alldepth occlusion images are filed separately for each encoded stream,when a plurality of demanded depth occlusion images are acquired, it isnecessary to acquire quality information from a plurality of files, andthe acquisition efficiency is low.

It is to be noted that, while the acquisition unit 81 does not acquiremetadata including quality information of texture images of 8 Mbps and 4Mbps, otherwise the metadata may be acquired. In this case, the segmentfile generation unit 83 generates also a segment file in which encodedstreams of metadata including quality information of the texture imagesof 8 Mbps and 4 Mbps are stored collectively in a unit of a segment.Further, metadata of the texture image of 8 Mbps and metadata of thetexture image of 4 Mbps are disposed in tracks different from eachother.

(Example of Description of QualityMetricsSampleEntry)

FIG. 6 is a view depicting an example of a description ofQualityMetricsSampleEntry disposed in a quality file.

As depicted in FIG. 6, in QualityMetricsSampleEntry,QualityMetricsConfigurationBox is disposed. In theQualityMetricsConfigurationBox, field_size_bytes and metric_count aredescribed and the number of metric_count equal to metric_code aredescribed.

As depicted in FIG. 7, field_size_bytes indicates a data size per onekind of quality (Quality) of an encoded stream of quality informationincluded in a sample of a quality file. In the case where the actualsize of an encoded stream of a certain kind of quality information issmaller than field_size_bytes, padding is added to the encoded stream ofthe quality information.

Further, metric_count indicates the number of kinds of qualitycorresponding to the encoded stream of quality information included in asample of the quality file. metric_code is information representative ofeach kind of quality corresponding to the encoded stream of qualityinformation included in a sample of the quality file and is described inorder of the encoded stream of quality information disposed in thesample.

(Example of Metric_Code)

FIG. 8 is a view depicting an example of metric_code.

As depicted in FIG. 8, as metric_code, not only psnr, ssim, msim, j144,j247, mops, and fsig defined in ISO/IEC 23001-10 but also ocer and ocprcan be set.

For example, psnr represents that the kind of quality represented by thequality information is PSNR (Peak signal-to-noise ratio) of the entirescreen.

Further, ocer and ocpr are set when quality information of an occlusionimage is included in the sample. ocer represents that the kind ofquality represented by quality information indicates an occlusion regioncorresponding to the occlusion image with respect to the entire screenof the texture image, namely, a ratio of an effective range of theocclusion region. ocpr represents that the kind of quality representedby the quality information is PSNR only of an occlusion image, namely,PSNR only of an effective range of the occlusion region.

As described above, in the case where quality information of theocclusion image is included in the sample, ocer or ocpr can be set asmetric_code. Accordingly, the ratio of the occlusion region and qualityinformation representative of the PSNR can be stored in the sample.Therefore, the video reproduction terminal 14 can select and reproducean optimum occlusion file on the basis of the quality information.

In particular, since the occlusion image is an image only of anocclusion region within the screen, there is the possibility that theinfluence on existing quality such as PSNR of the entire screen may beless. Accordingly, quality of an occlusion image cannot be representedsufficiently by existing quality information. Therefore, by making itpossible to store quality information representing, as quality, a ratioor a PSNR of the occlusion region having a great influence on quality ofthe occlusion image into the sample, it is made possible for the videoreproduction terminal 14 to select a more suitable occlusion file on thebasis of quality information.

(Example of Representation Element)

FIG. 9 is a view depicting an example of a representation element of anMPD file generated by the MPD file generation unit 84 of FIG. 4.

As depicted in FIG. 5, the segment file generation unit 83 generatesseven kinds of segment files including a texture file (texture1 file),another texture file (texture2 file), a depth file (depth1 file),another depth file (depth2 file), an occlusion file (occlusion1 file), aquality file (quality1 file(depth)), and another quality file (quality2file (occlusion)). Accordingly, as depicted in FIG. 9, sevenrepresentation elements are included in the MPD file.

(Example of Description of MPD File)

FIG. 10 is a view depicting an example of a description of an MPD filegenerated by the MPD file generation unit 84 of FIG. 4.

It is to be noted that, in the present specification, it is assumed thatthe picture quality of a 3D image is not improved even if the depth file(depth1 file) or the occlusion file (occlusion file) is used uponreproduction of the texture file (texture2 file). Accordingly,reproduction of a 3D image for which the texture file (texture2 file)and the depth file (depth1 file) or the occlusion file (occlusion file)are used is not to be performed. Therefore, reproduction patterns 1 to 7described below are patterns to be reproduced. Reproduction pattern 1.Reproduction of a texture image of the texture file (texture1 file)Reproduction pattern 2. Reproduction of a texture image of the texturefile (texture2 file) Reproduction pattern 3. Reproduction of a 3D imagefor which the texture file (texture1 file) and the depth file (depth1file) are used Reproduction pattern 4. Reproduction of a 3D image forwhich the texture file (texture1 file) and the depth file (depth2 file)are used Reproduction pattern 5. Reproduction of a 3D image for whichthe texture file (texture1 file), depth file (depth1 file), andocclusion file (occlusion file) are used Reproduction pattern 6.Reproduction of a 3D image for which the texture file (texture1 file),depth file (depth2 file), and occlusion file (occlusion file) are used

Reproduction pattern 7. Reproduction of a 3D image for which the texturefile (texture2 file) and the depth file (depth2 file) are used

In the MPD file of FIG. 10, a texture file (texture1 file) group andanother texture file (texture2 file) group are grouped by one adaptationset (Adaptation Set).

In the adaptation set element for the texture file, a representationelement corresponding to the texture file (texture1 file) group andanother representation element corresponding to the texture file(texture2 file) group are described.

The representation element includes Representation id, bandwidth,BaseURL, associationID and so forth. Representation id is an ID uniqueto the representation element and is information for specifying anencoded stream corresponding to the representation element. bandwidth isinformation representing a bit rate of the texture file group, andBaseURL is information representing a base of a file name. Further,associationID is Representation id of some other representation elementrelating to decoding or displaying (reproduction). This associationID isprescribed by ISO/IEC 23009-1 Amendment2.

Accordingly, in the representation element corresponding to the texturefile (texture1 file) group, vt1 is described as Representation id,8192000 representing 8 Mbps as bandwidth, and “texture1.mp4” as BaseURL.

In the representation element corresponding to the texture file(texture2 file) group, vt2 is described as Representation id, 4096000representing 4 Mbps as bandwidth, and “texture2.mp4” as BaseURL.

It is to be noted that, since the representation elements correspondingto the texture file (texture1 file) group and the texture file (texture2file) group do not relate to the other representation elements,associationID is not described in the representation elementscorresponding to the texture file (texture1 file) group and the texturefile (texture2 file) group.

Further, the depth file (depth1 file) group and the depth file (depth2file) group are grouped by one adaptation set element. In the adaptationset element for the depth files, a representation element correspondingto the depth file (depth1 file) group and another representation elementcorresponding to the depth file (depth2 file) group are described.

In the representation element corresponding to the depth file (depth1file), vd1 is described as Representation id, 2048000 representing 2Mbps as bandwidth, and “depth1.mp4” as BaseURL.

Further, in the reproduction patterns 1 to 7, the texture file groupthat relates to the depth file (depth1 file) is the texture file(texture1 file) group. Accordingly, in the representation elementcorresponding to the depth file (depth1 file) group, vt1 that isRepresentation id of the texture file (texture1 file) group is describedas associationID.

In the representation element corresponding to the depth file (depth2file), vd2 is described as Representation id, 1024000 representing 1Mbps as bandwidth, and “depth2.mp4” as BaseURL.

Further, in the reproduction patterns 1 to 7, a texture file group of atexture image relating to the depth file (depth2 file) upon displayingis the texture file (texture1 file) group or the texture file (texture2file) group. Accordingly, in the representation element corresponding tothe depth file (depth1 file) group, vt1 that is Representation id of thetexture file (texture1 file) group and vt2 that is Representation id ofthe texture file (texture2 file) group are described as associationID.

Further, the occlusion file (occlusion1 file) group is grouped with oneadaptation set element. In the adaptation set element for the occlusionfile, a representation element corresponding to the occlusion file(occlusion file) group is described.

In the representation element corresponding to the occlusion file(occlusion file) group, vo1 is described as Representation id, 1024000representing 1 Mbps as bandwidth, and “occlusion.mp4” as BaseURL.

Further, in the reproduction patterns 1 to 7, a depth file group of thedepth image relating to the occlusion file (occlusion file) upondisplaying is the depth file (depth1 file) group or the depth file(depth2 file) group. Accordingly, in the representation elementcorresponding to the occlusion file (occlusion file) group, vd1 that isRepresentation id of the depth file (depth1 file) group and vd2 that isRepresentation id of the depth file (depth2 file) group are described asassociationID.

Further, the quality file (quality1 file) group and the quality file(quality2 file) group are grouped by one adaptation set element.

In the adaptation set element for the quality file,schemeIdUri=“urn:mpeg:dash:quality:playback:combination:2015” can bedescribed which is schemeIdUri for representing a combination of imagesto be reproduced from among the texture image, the depth image, and theocclusion image, namely, a combination of images that are made usecandidates upon reproduction, using SupplementalProperty. Representationid of imaged configuring the combination is described as value ofschemeIdUri=“urn:mpeg:dash:quality:playback:combination:2015.”

In particular, as information representative of a texture image to beused for reproduction of the reproduction pattern 1 described above,<SupplementalPropertyschemeldUri=“urn:mpeg:dash:quality:playback:combination:2015value=‘vt1’”> in which vt1 that is Representation id of the texture file(texture1 file) is described.

Similarly, as information representative of a texture image and a depthimage to be used for reproduction of the reproduction pattern 3,<SupplementalPropertyschemeIdUri=“urn:mpeg:dash:quality:playback:combination:2015 value=‘vt1vd1’”> in which vt1 that is Representation id of the texture file(texture1 file) and vd1 that is Representation id of the depth file(depth1 file) is value is described.

As information representing a texture image and a depth image to be usedfor reproduction of the reproduction pattern 4, <SupplementalPropertyschemeIdUri=“urn:mpeg:dash:quality:playback:combination:2015 value=‘vt1vd2’”> in which vt1 that is Representation id of the texture file(texture1 file) and vd2 that is Representation id of the depth file(depth2 file) are value is described.

As information representative of a texture image, a depth image, and anocclusion image used for reproduction in the reproduction pattern 5,<SupplementalPropertyschemeIdUri=“urn:mpeg:dash:quality:playback:combination:2015 value=‘vt1vd1 vo1’”> in which vt1 that is Representation id of the texture file(texture1 file), vd1 that is Representation id of the depth file (depth1file), and vo1 that is Representation id of the occlusion file(occlusion file) are value is described.

As information representative of a texture image, a depth image, and anocclusion image used for reproduction in the reproduction pattern 6,<SupplementalPropertyschemeIdUri=“urn:mpeg:dash:quality:playback:combination:2015 value=‘vt1vd2 vo1’”> in which vt1 that is Representation id of the texture file(texture1 file), vd2 that is Representation id of the depth file (depth2file), and vo1 that is Representation id of the occlusion file(occlusion file) are value is described.

As information representative a texture image used for reproduction inthe reproduction pattern 2, <SupplementalPropertyschemeIdUri=“urn:mpeg:dash:quality:playback:combination:2015”> in whichvt2 that is Representation id of the texture file (texture1 file) isvalue is described.

As information representing a texture image and a depth image used forreproduction in the reproduction pattern 7, <SupplementalPropertyschemeIdUri=“urn:mpeg:dash:quality:playback:combination:2015”> in whichvt2 that is Representation id of the texture file (texture2 file) andvd2 that is Representation id of the depth file (depth2 file) are valueis described.

As described above, Representation id of images that configure acombination of images used in the pattern to be reproduced is describedin the MPD file. Accordingly, the video reproduction terminal 14 canrefer to the MPD file to perform reproduction such that reproduction isnot performed with any other pattern than the pattern to be reproduced.

Further, in the adaptation set element for the quality file,representation elements individually corresponding to the quality file(quality1 file) group and the quality file (quality2 file) group aredescribed.

In the representation element corresponding to the quality file(quality1 file(depth)) group, vq1 is described as Representation id and“quality1.mp4” is described as BaseURL.

Further, quality information stored in the quality file (quality1 file)is quality information of depth images stored in the depth file (depth1file) and the depth file (depth2 file). Accordingly, in therepresentation element corresponding to the quality file (quality1 file)group, vd1 that is Representation id of the depth file (depth1 file) andvd2 that is Representation id of the depth file (depth2 file) aredescribed as associationID.

Consequently, the video reproduction terminal 14 can recognize thatquality information stored in the quality file (quality1 file) group isquality information of depth images corresponding to the depth file(depth1 file) group and the depth file (depth2 file) group.

However, the video reproduction terminal 14 cannot recognize whether thequality information stored in one of two tracks of the quality file(quality1 file) is quality information of depth images stored in thedepth file (depth1 file) group or the depth file (depth2 file) group.

Accordingly, in the MPD file of FIG. 10, a sub representation(SubRepresentation) element obtained by dividing a representationelement for each level that can be associated with a track is expandedsuch that the sub representation element can have associationIDsimilarly to the Representation element. Consequently, a correspondingrelationship between the respective tracks of the quality file (quality1file) and RepresentationID (depth-related image specificationinformation) for specifying a depth image can be described.

In particular, in the example of FIG. 10, a track that stores qualityinformation of a depth image to be stored into the depth file (depth1file) is associated with the level 1 by a leva box (LevelAssignmentBox)disposed in the quality file (quality1 file). Further, a track thatstores quality information of a depth image to be stored into the depthfile (depth2 file) is associated with the level 2.

Accordingly, <SubRepresentation level=“1” associationID=“vd1”> thatassociates the level 1 and vd1 that is Representation id of the depthfile (depth1 file) as associationID is described. Further,<SubRepresentation level=“2” associationID=“vd2”> that associates thelevel 2 and vd2 that is Representation id of the depth file (depth2file) as associationID with each other is described.

Further, in the representation element corresponding to the quality file(quality2 file) group, vq2 is described as Representation id and“quality2.mp4” is described as BaseURL.

Quality information included in metadata stored in the quality file(quality2 file) is quality information of an occlusion image stored inthe occlusion file (occlusion file). Accordingly, in the representationelement corresponding to the quality file (quality1 file) group, vo1that is Representation id of the occlusion file (occlusion file) groupis described as associationID.

It is to be noted that, while, in the example of FIG. 10, bandwidth isnot described in the adaptation set element for a quality file, it mayotherwise be described.

(Description of Leva Box)

FIG. 11 is a view illustrating a leva box disposed in the quality file(quality1 file) in the case where the MPD file is the MPD file of FIG.10.

As depicted in FIG. 11, the leva box (Level assignment box) is disposedin the quality file (quality1 file) that has a plurality of tracks. Inthis leva box, information that specifies a track corresponding to eachof levels is described in order beginning with the level 1 to describe acorresponding relationship between the levels and the tracks.

Accordingly, by the leva box, the video reproduction terminal 14 canspecify a track corresponding to associationID a sub representationelement of the MPD file has. In particular, the video reproductionterminal 14 can specify the track corresponding to associationID that isvd1 to the first track (quality track(depth1)) from the topcorresponding to the level 1. Further, the video reproduction terminal14 can specify the track corresponding to associationID that is vd2 tothe second track (quality track(depth2)) from the top corresponding tothe level 2.

(Description of Processing of File Reproduction Apparatus)

FIG. 12 is a flow chart illustrating a file generation process of thefile generation apparatus 11 of FIG. 1.

At step S11 of FIG. 12, the acquisition unit 81 of the file generationapparatus 11 acquires image data of a texture image, a depth image, andan occlusion image of a video content and metadata including qualityinformation of encoded streams of depth images of 2 Mbps and 1 Mbps andan occlusion image of 1 Mbps. Then, the acquisition unit 81 supplies theacquired image data and metadata to the encoding unit 82.

At step S12, the encoding unit 82 encodes the image data of the textureimage supplied from the acquisition unit 81 in 8 Mbps and 4 Mbps andencodes the image data of the depth image in 2 Mbps and 1 Mbps. Further,the encoding unit 82 encodes the image data of the occlusion image in 1Mbps. Further, the encoding unit 82 encodes the metadata of the depthimages of 2 Mbps and 1 Mbps and the occlusion image of 1 Mbpsindividually at predetermined bit rates. The encoding unit 82 suppliesencoded streams generated as a result of the encoding to the segmentfile generation unit 83.

At step S13, the segment file generation unit 83 files the encodedstreams of the texture image, the depth image, and the occlusion imagesupplied from the encoding unit 82 in a unit of a segment for each bitrate. The segment file generation unit 83 supplies a texture file, adepth file, and an occlusion file generated as a result of the filing tothe upload unit 85.

At step S14, the segment file generation unit 83 divides the encodedstreams of the metadata supplied from the encoding unit 82 into two foreach of kinds of a depth occlusion image.

At step S15, the segment file generation unit 83 disposes the divisionsof the encoded streams of the metadata into quality files different fromeach other in a unit of a segment to generate and supply a quality fileto the upload unit 85.

At step S16, the MPD file generation unit 84 generates and supplies anMPD file to the upload unit 85. At step S17, the upload unit 85 uploadsthe texture file, the depth file, the occlusion file, the quality file,and the MPD file to the Web server 12.

As described above, the file generation apparatus 11 divides qualityinformation of a depth occlusion image for each kind of depth occlusionimage and disposes the divisions of the divided information into qualityfiles different from each other. Accordingly, in comparison with analternative case in which quality information is disposed into qualityfiles different from each other for each depth occlusion image, thenumber of quality files can be reduced. Accordingly, it can be said thatthe quality information of the depth occlusion image can be storedefficiently. Further, the processing amount relating to acquisition ofquality information by the video reproduction terminal 14 can bereduced.

Further, where the texture image and the depth image are used forreproduction, the video reproduction terminal 14 can acquire qualityinformation from the quality file (quality1 file) in which only thequality information of the depth image is stored. Accordingly, theacquisition efficiency of quality information can be increased incomparison with that in an alternative case in which the qualityinformation is acquired from the quality file in which qualityinformation of all depth occlusion images is stored.

Further, the file generation apparatus 11 generates an MPD file in whichassociationId is described in the sub representation element.Consequently, the MPD file can manage a corresponding relationshipbetween tracks of the quality file in which quality information of aplurality of depth occlusion images is divisionally disposed in tracksdifferent from each other and the depth occlusion image. As a result,the video reproduction terminal 14 can extract quality information ofthe depth occlusion images from the quality file in which the qualityinformation of a plurality of depth occlusion images is divisionallydisposed in tracks different from each other.

Further, since the file generation apparatus 11 generates an MPD file inwhich a combination of images to be used for reproduction of a patternto be used for reproduction is described, only reproduction of thepattern to be used for reproduction can be performed by the videoreproduction terminal 14. As a result, for example, a producer of thevideo content can provide an image of quality desired by the producer tothe user. Further, since only it is necessary for the video reproductionterminal 14 to select a reproduction pattern from the patterns to beused for reproduction, the processing load is reduced in comparison withthat in an alternative case in which a reproduction pattern is selectedfrom among all reproduction patterns capable of being used forreproduction.

(Example of Functional Configuration of Video Reproduction Terminal)

FIG. 13 is a block diagram depicting an example of a configuration of astreaming reproduction unit implemented by the video reproductionterminal 14 of FIG. 1 executing the controlling software 21, the videoreproduction software 22, and the accessing software 23.

The streaming reproduction unit 100 is configured from an MPDacquisition unit 101, an MPD processing unit 102, a quality informationacquisition unit 103, a decoding unit 104, an image acquisition unit105, a decoding unit 106, an output controlling unit 107, and ameasurement unit 108.

The MPD acquisition unit 101 of the streaming reproduction unit 100requests the Web server 12 for acquires an MPD file. The MPD acquisitionunit 101 supplies the acquired MPD file to the MPD processing unit 102.

The MPD processing unit 102 analyzes the MPD file supplied from the MPDacquisition unit 101. In particular, the MPD processing unit 102acquires bandwidth each Representation element of the MPD file has as abit rate of an image corresponding to the Representation element.

Further, the MPD processing unit 102 acquires a combination of images tobe used for reproduction of the pattern to be used for reproduction fromvalue of schemeIdUri=“urn:mpeg:dash:quality:playback:combination:2015”of the MPD file and Representation id each Representation element has.Furthermore, the MPD processing unit 102 acquires acquisitioninformation such as a file name of a segment file group corresponding tothe Representation element, a level corresponding to quality informationof the respective depth occlusion images and so forth from BaseURL eachRepresentation element of the MPD file has, associationId of the subrepresentation element and so forth.

The MPD processing unit 102 selects candidates for a reproductionpattern from among the patterns to be used for reproduction. The MPDprocessing unit 102 creates a list of acquisition candidates for a depthocclusion image on the basis of the network bandwidth of the Internet 13and the bit rate of the image supplied from the measurement unit 108.The MPD processing unit 102 supplies acquisition information of qualityinformation of the depth occlusion image registered in the list to thequality information acquisition unit 103.

For example, in the case where a depth image is registered in the listof acquisition candidates for a depth occlusion image, the MPDprocessing unit 102 supplies acquisition information of qualityinformation of the depth image registered in the list in the qualityfile (quality1 file) to the quality information acquisition unit 103. Inthe case where an occlusion image is registered in the list ofacquisition candidates for a depth occlusion image, the MPD processingunit 102 supplies acquisition information of quality information of theocclusion image registered in the list in the quality file (quality2file) to the quality information acquisition unit 103.

Further, the MPD processing unit 102 selects a reproduction pattern fromcandidates for a reproduction pattern on the basis of the qualityinformation supplied from the decoding unit 104. The MPD processing unit102 supplies the acquisition information of the texture image of thetexture file to be used for reproduction of the selected reproductionpattern to the image acquisition unit 105. Further, where a file of thedepth occlusion image is used for reproduction of the selectedreproduction pattern, the MPD processing unit 102 supplies theacquisition information of the depth occlusion image to the imageacquisition unit 105.

The quality information acquisition unit 103 requests the Web server 12for and acquires an encoded stream of metadata including the qualityinformation on the basis of the acquisition information supplied fromthe MPD processing unit 102. The quality information acquisition unit103 supplies the acquired encoded stream to the decoding unit 104.

The decoding unit 104 decodes the encoded stream supplied from thequality information acquisition unit 103 and generates metadataincluding the quality information. The decoding unit 104 supplies thequality information to the MPD processing unit 102.

The image acquisition unit 105, the decoding unit 106, and the outputcontrolling unit 107 function as a reproduction unit, and reproduce onlythe texture image or the texture image and the depth occlusion image onthe basis of the acquired information supplied from the MPD processingunit 102.

In particular, the image acquisition unit 105 requests the Web server 12for and acquires an encoded stream of a texture file or a file of thedepth occlusion image on the basis of the acquisition informationsupplied from the MPD processing unit 102. The image acquisition unit105 supplies the acquired encoded stream to the decoding unit 104.

The decoding unit 106 decodes the encoded stream supplied from the imageacquisition unit 105 to generate image data. The decoding unit 106supplies the generated image data to the output controlling unit 107.

In the case where the image data supplied from the decoding unit 106 isonly image data of the texture image, the output controlling unit 107causes a display unit not depicted such as a display the videoreproduction terminal 14 has to display the texture image thereon on thebasis of the image data of the texture image.

On the other hand, in the case where the image data supplied from thedecoding unit 106 is image data of the texture image and the depthocclusion image, the output controlling unit 107 generates image data ofa 3D image on the basis of the image data of the texture image and thedepth occlusion image. Then, the output controlling unit 107 causes thedisplaying unit not depicted such as a display to display a 3D image onthe basis of the generated image data of the 3D image.

The measurement unit 108 measures the network bandwidth of the Internet13 and supplies the measured network bandwidth to the MPD processingunit 102.

(Description of First Example of Process of Streaming Reproduction Unit)

FIG. 14 is a flow chart illustrating a first example of a reproductionprocess of the streaming reproduction unit 100 of FIG. 13. It is to benoted that, in the reproduction process of FIG. 14, the streamingreproduction unit 100 performs reproduction of a 3D image that uses atexture image and a depth image.

At step S31 of FIG. 14, the MPD acquisition unit 101 requests the Webserver 12 for and acquires an MPD file. The MPD acquisition unit 101supplies the acquired MPD file to the MPD processing unit 102.

At step S32, the MPD processing unit 102 analyzes the MPD file suppliedfrom the MPD acquisition unit 101. consequently, the MPD processing unit102 acquires bit rates of the respective texture images and depthimages, a combination of images to be used for reproduction in a patternto be reproduced and acquisition information of a texture image, a depthimage, and quality information.

At step S33, the MPD processing unit 102 selects, on the basis of thecombination of images to be used for reproduction in the pattern to bereproduced, the reproduction patterns 3, 4, and 7 in which reproductionis performed using only a texture image and a depth image from among thepatterns to be reproduced as candidates for a reproduction pattern.Processes at succeeding steps S34 to S43 are performed in a unit of asegment.

At step S34, the measurement unit 108 measures and supplies a networkbandwidth of the Internet 13 to the MPD processing unit 102.

At step S35, the MPD processing unit 102 determines, on the basis of thenetwork bandwidth and the bit rate of the texture images, a textureimage to be acquired from among the texture images to be used forreproduction with the candidates for a reproduction pattern.

For example, the MPD processing unit 102 assumes that 80 percent of thenetwork bandwidth is a maximum acceptable bit rate for a texture imageand determines a texture image of a bit rate lower than the maximumacceptable bit rate from among the texture images to be used forreproduction with the candidates for a reproduction pattern as a textureimage to be acquired.

At step S36, the MPD processing unit 102 creates a list of acquisitioncandidates for a depth image on the basis of the candidates for areproduction pattern for reproducing a texture image to be acquired andthe bit rate of the depth image.

For example, the MPD processing unit 102 determines that 20 percent ofthe network bandwidth is a maximum acceptable bit rate for a depthimage. Then, in the case where the texture image to be acquired is atexture image of a texture file (texture1 file) and the bit rates ofdepth images of the depth file (depth1 file) and the depth file (depth2file) are lower than the maximum acceptable bit rate, the MPD processingunit 102 creates a list in which the depth images of the depth file(depth1 file) and the depth file (depth2 file) are registered on thebasis of the reproduction patterns 3 and 4.

On the other hand, in the case where the texture image to be acquired isa texture image of the texture file (texture2 file) and the bit rates ofdepth images of the depth file (depth2 file) and the depth file (depth2file) are lower than the maximum acceptable bit rate, the MPD processingunit 102 creates a list in which the depth image of the depth file(depth2 file) is registered on the basis of the reproduction pattern 7.

Then, the MPD processing unit 102 supplies acquisition information ofquality information of the depth images registered in the list to thequality information acquisition unit 103. It is to be noted that, in thecase where all bit rates of the depth images to be reproduced togetherwith the texture images to be acquired are equal to or higher than themaximum acceptable bit rate, nothing is registered into the list ofacquisition candidates for a depth image, and only encoded streams oftexture images are acquired, decoded, and displayed, and the processadvances to step S43.

At step S37, the quality information acquisition unit 103 requests, onthe basis of the acquisition information supplied from the MPDprocessing unit 102, the Web server 12 for an encoded stream of metadataincluding quality information of the depth images to acquire the encodedstream. The quality information acquisition unit 103 supplies theacquired encoded stream to the decoding unit 104.

At step S38, the decoding unit 104 decodes the encoded stream of thequality information of the depth images supplied from the qualityinformation acquisition unit 103 to generate metadata including thequality information of the depth images. The decoding unit 104 suppliesthe quality information of the depth images to the MPD processing unit102.

At step S39, the MPD processing unit 102 determines a depth image to beacquired from among the depth images registered in the list of depthimages on the basis of the quality information supplied from thedecoding unit 104.

For example, the MPD processing unit 102 determines a depth image whosequality represented by the quality information is best, a depth imagewhose quality represented by the quality information is nearest to thequality of a depth image of an immediately preceding segment (or subsegment), or a depth image whose quality represented by the qualityinformation is acceptable quality and besides the bit rate is lowest asa depth image to be acquired.

In the case where a depth image whose quality represented by the qualityinformation is nearest to the quality of a depth image of an immediatelypreceding segment (or sub segment) is determined as a depth image to beacquired, the sense of incongruity of the appearance of the 3D image tobe reproduced can be reduced. The MPD processing unit 102 suppliesacquisition information of the depth image to be acquired to the imageacquisition unit 105.

At step S40, the image acquisition unit 105 requests the Web server 12for an encoded stream of the texture image and the depth image on thebasis of the acquisition information supplied from the MPD processingunit 102 to acquire encoded streams. The image acquisition unit 105supplies the acquired encoded streams to the decoding unit 104.

At step S41, the encoded streams supplied from the image acquisitionunit 105 are decoded to generate image data of the texture image and thedepth image. The decoding unit 106 supplies the generated image data ofthe texture image and the depth image to the output controlling unit107.

At step S42, the output controlling unit 107 generates image data for a3D image on the basis of the image data of the texture image and thedepth image supplied from the decoding unit 106 and controls the displayunit not depicted to display a 3D image.

At step S43, the streaming reproduction unit 100 decides whether animage of the last segment of the video content is displayed. In the casewhere it is decided at step S43 that an image of the last segment of thevideo content is not displayed as yet, the process returns to step S34.

On the other hand, in the case where it is decided at step S43 that animage of the last segment of the video content is displayed, the processends.

It is to be noted that, though not depicted, the first example of thereproduction process by which reproduction of a 3D image is performed bythe streaming reproduction unit 100 using the texture image, the depthimage, and the occlusion image is similar to the reproduction process ofFIG. 14 except the following point.

In particular, the candidates for a reproduction pattern selected atstep S33 of FIG. 14 are the reproduction patterns 5 and 6. Further,between step S39 and step S40, a process regarding quality informationof the occlusion image is performed similarly to the processes at stepsS36 to S39 for the quality information of the depth image.

However, in this case, the maximum acceptable bit rates for a textureimage, a depth image, and an occlusion image are 70 percent, 15 percent,and 15 percent of the network bandwidth, respectively. Furthermore, thedepth image in the processes at steps S40 to S42 is both a depth imageand an occlusion image.

(Description of Second Example of Processing of Streaming ReproductionUnit)

FIG. 15 is a flow chart illustrating a second example of thereproduction process of the streaming reproduction unit 100 of FIG. 13.It is to be noted that, in the reproduction process of FIG. 15, thestreaming reproduction unit 100 performs reproduction of a 3D image inwhich a texture image and a depth image are used.

The reproduction process of FIG. 15 is different from the reproductionprocess of FIG. 14 in that the ratio of the maximum acceptable bit ratesof the texture image and the depth image to the network bandwidth is notdetermined.

Steps S61 to S64 of FIG. 15 are similar to the processes at steps S31 toS34 of FIG. 14, respectively, and therefore, description of them isomitted. Processes at steps S64 to S73 are executed in a unit of asegment.

At step S65, the MPD processing unit 102 creates a list of acquisitioncandidates for a combination of a texture image and a depth image on thebasis of the candidates for a reproduction pattern, network bandwidth ofthe Internet 13 supplied from the measurement unit 108, and bit rate ofthe texture image and the depth image.

In particular, a list is created in which, from among combinations oftexture images and depth images used for reproduction in thereproduction patterns 3, 4, and 7, combinations in which the sum of bitrates of the texture image and the depth image does not exceed thenetwork bandwidth are registered.

It is to be noted that a lower limit for the bit rates of the textureimages and the depth images may be determined in advance such that, fromamong the combinations registered in the list, any combination in whichat least one of the bit rates is lower than the lower limit is excluded.

Further, in the case where all of the sums of the bit rates of thetexture images and the depth images that are used for reproduction inthe reproduction patterns 3, 4, and 7 exceed the network bandwidth,nothing is registered into the list of acquisition candidates for acombination of a texture image and a depth image. Then, only the encodedstream of a texture image of a maximum bit rate that does not exceed thenetwork bandwidth is acquired, decoded, and displayed, and the processadvances to step S73.

At step S66, the MPD processing unit 102 creates a list of the depthimages registered in the list created at step S65. Then, the MPDprocessing unit 102 supplies acquisition information of qualityinformation of the depth images registered in the list to the qualityinformation acquisition unit 103.

Processes at steps S67 and S68 are similar to the processes at steps S37and S38 of FIG. 14, respectively, and therefore, description of them isomitted.

At step S69, the MPD processing unit 102 determines, on the basis of thequality information, a combination of a texture image and a depth imageto be acquired from among the combinations registered in the list ofcombinations of texture images and depth images.

For example, the MPD processing unit 102 determines a depth image to beacquired similarly as at step S39 of FIG. 14. Then, the MPD processingunit 102 determines a texture image of the highest bit rate from amongthe texture images whose combination with the depth image to be acquiredis registered in the combination list as a texture image to be acquired.

Processes at steps S70 to S73 are similar to the processes at steps S40to S43 of FIG. 14, respectively, and therefore, description of them isomitted.

It is to be noted that, though not depicted, the second example of thereproduction process in which the streaming reproduction unit 100executes reproduction of a 3D image using a texture image, a depthimage, and an occlusion image is similar to the reproduction process ofFIG. 15 except the following point.

In particular, the candidates for a reproduction pattern selected atstep S63 of FIG. 15 are the reproduction patterns 5 and 6. Further, thedepth image in the processes at steps S65 to S72 includes both a depthimage and an occlusion image.

Since the video reproduction terminal 14 acquires quality information ofdepth and occlusion images as described above, the video reproductionterminal 14 can acquire appropriate depth occlusion images on the basisof the quality information.

Second Embodiment (Example of Segment File)

The configuration of a second embodiment of the information processingsystem to which the present disclosure is applied is same as theconfiguration of the information processing system 10 of FIG. 1principally except that an encoded stream of metadata including qualityinformation is divided not for each kind of a depth-related image butfor each texture image and is disposed into a different one of qualityfiles. Accordingly, in the following description, description other thandescription of a quality file is omitted suitably.

FIG. 16 depicts an example of a segment file generated by the segmentfile generation unit 83 of the second embodiment of the informationprocessing system to which the present disclosure is applied.

The segment file of FIG. 16 is same as the segment file of FIG. 5 excepta quality file.

As depicted in FIG. 16, the segment file generation unit 83 divides eachof encoded streams of quality information of depth images of 2 Mbps and1 Mbps and encoded streams of quality information of an occlusion imageof 1 Mbps into two for each texture image to be reproduced with theencoded streams. Then, the segment file generation unit 83 disposes thedivisions of the encoded streams of metadata in a unit of a segment intothe different quality files to generate quality files.

In particular, a depth occlusion image used for reproduction togetherwith a texture file (texture1 file) in reproduction with a pattern to bereproduced is depth images of 2 Mbps and 1 Mbps and an occlusion imageof 1 Mbps. Accordingly, the segment file generation unit 83 generates aquality file (quality1 file) in which encoded streams of metadataincluding quality information of the depth images of 2 Mbps and 1 Mbpsand encoded streams of metadata including quality information of theocclusion image of 1 Mbps are disposed in a unit of a segment.

In the quality file (quality1 file), the respective encoded streams aredisposed in different tracks (quality track(depth1), qualitytrack(depth2), and quality track(occlusion1)).

Further, a depth occlusion image used for reproduction together with thetexture file (texture2 file) in reproduction of a pattern to bereproduced is a depth image of 1 Mbps. Accordingly, the segment filegeneration unit 83 generates the quality file (quality2 file) in whichthe encoded stream of metadata including quality information of thedepth image of 1 Mbps is displayed in a unit of a segment.

In this manner, the segment file generation unit 83 files encodedstreams of metadata including quality information separately into filesfor each texture image. Accordingly, by acquiring quality informationfrom a quality file of texture images to be acquired, the videoreproduction terminal 14 can readily acquire quality information of adepth occlusion image to be used for reproduction in a pattern to bereproduced together with the texture image.

In particular, in a quality file, quality information of a depthocclusion image to be used in reproduction in a pattern to be reproducedtogether with a texture image corresponding to the quality file isstored. Accordingly, in comparison with an alternative case in whichencoded streams of quality information of all depth occlusion images arecollectively filed, the quality information of depth occlusion images tobe used for reproduction together with texture images to be acquired canbe acquired readily.

Further, though not detected, the MPD file in the second embodiment issimilar to the MPD file of FIG. 10 except the following point. Inparticular, in the MPD file in the second embodiment, associationID therepresentation element of the quality file (quality1 file) has includesnot only vd1 and vd2 but also vo1. Further, the representation elementof the quality file (quality1 file) further includes a subrepresentation element having the level 3 and vo1 as associationID.Furthermore, associationID the representation element of the qualityfile (quality2 file) has is not vo1 but vd2.

Further, though not depicted, the file reproduction process in thesecond embodiment is same as the file reproduction process of FIG. 12except that an encoded stream of metadata including quality informationis divided not for each kind of a depth-related image but for eachtexture image at step S14.

Further, though not depicted, at step S36, the reproduction process inthe second embodiment is same as the reproduction process of FIG. 14except that the MPD processing unit 102 supplies acquisition informationof quality information stored in a quality file of texture images to beacquired from within the acquisition information of the qualityinformation of depth images registered in the list to the qualityinformation acquisition unit 103.

As described above, the file generation apparatus 11 in the secondembodiment divides quality information of a depth occlusion image foreach texture image to be reproduced together with the depth occlusionimage and disposes the divisions of the quality information intodifferent quality files. Accordingly, in comparison with an alternativecase in which quality information is disposed in different quality filesfor different depth occlusion images, the number of quality files can bereduced. Therefore, it can be said that the quality information of depthocclusion images can be stored efficiently. Further, the processingamount relating to acquisition of the quality information by the videoreproduction terminal 14 can be reduced.

Further, the video reproduction terminal 14 can acquire, from within aquality file in which quality information only of a depth occlusionimage to be reproduced together with a texture image of a reproductiontarget is stored, the quality information. Accordingly, in comparisonwith an alternative case in which quality information is acquired from aquality file in which quality information of all depth occlusion imagesis stored, the acquisition efficiency of quality information can beimproved.

Third Embodiment (Example of Segment File)

The configuration of a third embodiment of the information processingsystem to which the present disclosure is applied is same as theconfiguration of the information processing system 10 of FIG. 1principally except that the quality information of a depth occlusionimage is replaced by the quality information of a texture image or a 3Dimage reproduced in a pattern to be reproduced and that an encodedstream of quality information is divided for each kind of a reproductionpattern corresponding to the quality information and the divisions ofthe encodes stream are disposed in different quality files. Accordingly,in the following, description other than the description of a qualityfile is omitted suitably.

FIG. 17 is a view depicting an example of a segment file generated bythe segment file generation unit 83 in the third embodiment of theinformation processing system to which the present disclosure isapplied.

The segment file of FIG. 17 is same as the segment file of FIG. 5 exceptthe quality file.

As depicted in FIG. 17, the segment file generation unit 83 divides anencoded stream of metadata including quality information of patterns 1to 7 supplied from the encoding unit 82 into three for each kind of areproduction pattern. Then, the segment file generation unit 83 disposesthe divisions of the encoded stream in a unit of a segment intodifferent quality files to generate quality files.

In particular, the segment file generation unit 83 generates a qualityfile (quality1 file) in which encoded streams of the quality informationof the reproduction patterns 1 and 2 in which reproduction is performedusing only a texture image are disposed in a unit of a segment. In thequality file (quality1 file), the respective encoded streams aredisposed in different tracks (quality track(texture1) and qualitytrack(texture2)).

Further, the segment file generation unit 83 generates a quality file(quality2 file) in which encoded streams of quality information of thereproduction patterns 3, 4, and 7 in which reproduction is performedusing only a texture image and a depth image are disposed in a unit of asegment. In the quality file (quality2 file), the respective encodedstreams are disposed on different tracks (qualitytrack(texture1+depth1), quality track(texture1+depth2), and qualitytrack(texture2+depth2)).

Furthermore, the segment file generation unit 83 generates a qualityfile (quality3 file) in which encoded streams of metadata includingquality information of the reproduction patterns 5 and 6 in whichreproduction is performed using a texture image, a depth image, and anocclusion image are disposed in a unit of a segment. In the quality file(quality3 file), the respective encoded streams are disposed indifferent tracks (quality track(texture1+depth+occlusion1) and qualitytrack(texture1+depth2+occlusion1)).

As described above, the segment file generation unit 83 disposes anencoded stream of metadata including quality information of a textureimage or a 3D image to be reproduced in a pattern to be reproduced.Accordingly, the video reproduction terminal 14 can perform reproductionin which the quality of a texture image or a 3D image to be reproducedfinally is high on the basis of quality information.

Further, the segment file generation unit 83 files encoded streams ofquality information of a pattern to be reproduced separately for eachkind of the reproduction pattern. Accordingly, the video reproductionterminal 14 can readily acquire quality information of reproductionpatterns that become candidates from quality files of kinds ofreproduction patterns that are made candidates.

In contrast, in the case where encoded streams of quality information ofall reproduction patterns are filed collectively, it is necessary toacquire quality information of reproduction patterns that becomecandidates from files including also quality information of anunnecessary reproduction pattern that is not a kind of a reproductionpattern that becomes a candidate.

(Example of Description of MPD File)

FIG. 18 is a view depicting an example of a description of an MPD filein the third embodiment.

The description of the MPD file of FIG. 18 is same as the description ofFIG. 10 except the adaptation set element for a quality file.

In the MPD file of FIG. 18, a quality file (quality1 file) group,another quality file (quality2 file) group, and a further quality file(quality3 file) group are grouped by one adaptation set element.

In the adaptation set element for quality files, representation elementsindividually corresponding to the quality file (quality1 file) group,quality file (quality2 file) group, and quality file (quality3 file)group are described.

In the representation element corresponding to the quality file(quality1 file) group, vq1 is described as Representation id and“quality1.mp4” is described as BaseURL.

Meanwhile, quality information stored in the quality file (quality1file) is quality information of a texture file (texture1 file) andanother texture file (texture2 file). Accordingly, in a representationelement corresponding to the quality file (quality1 file), vt1 and vt2that are Representation id of the texture file (texture1 file) group andthe texture file (texture2 file) group are described as associationID.

Furthermore, in the example of FIG. 18, tracks corresponding to thelevel 1 and the level 2 have stored therein quality information of thetexture file (texture1 file) and the texture file (texture2 file),respectively.

Accordingly, in the representation element corresponding to the qualityfile (quality1 file) group, <SubRepresentation level=“1”associationID=“vt1”> that associates the level 1 and vt1 that isRepresentation id of the texture file (texture1 file) as associationIDis described. Further, <SubRepresentation level=“2” associationID=“vt2”>that associates the level 2 and vt2 that is Representation id of thetexture file (texture2 file) as associationID is described. In otherwords, a corresponding relationship between the respective tracks of thequality file (quality1 file) and Representation (texture imagespecification information) that specifies the texture images isdescribed.

Meanwhile, in the representation element corresponding to the qualityfile (quality2 file) group, vq2 is described as Representation id and“quality2.mp4” is described as BaseURL.

Further, quality information stored in the quality file (quality2 file)is quality information of a 3D image to be reproduced using the texturefile (texture1 file) and the depth file (depth1 file), another 3D imageto be reproduced using the texture file (texture1 file) and the depthfile (depth2 file), and a further 3D image to be reproduced using thetexture file (texture2 file) and the depth file (depth2 file).

Accordingly, in the representation element corresponding to the qualityfile (quality2 file) group, vt1, vt2, vd1, and vd2 that areRepresentation id of the texture file (texture1 file), the texture file(texture2 file), the depth file (depth1 file), and the depth file(depth2 file) are described as associationID.

Furthermore, in the example of FIG. 18, a track corresponding to thelevel 1 has stored therein quality information of a 3D image to bereproduced using the texture file (texture1 file) and the depth file(depth1 file). A track corresponding to the level 2 has stored thereinquality information of a 3D image to be reproduced using the texturefile (texture1 file) and the depth file (depth2 file). A trackcorresponding to the level 3 has stored therein quality information of a3D image to be reproduced using the texture file (texture2 file) and thedepth file (depth2 file).

Accordingly, in the representation element corresponding to the qualityfile (quality2 file) group, <SubRepresentation level=“1”associationID=“vt1 vd1”> that associates the level 1 and vt1 and vd1that are Representation id of the texture file (texture1 file) and thedepth file (depth1 file) as associationID with each other is described.Further, <SubRepresentation level=“2” associationID=“vt1 vd2”> thatassociates the level 2 and vt1 and vd2 that are Representation id of thetexture file (texture1 file) and the depth file (depth2 file) asassociationID with each other is described.

Furthermore, <SubRepresentation level=“3” associationID=“vt2 vd2”> thatassociates the level 3 and vt2 and vd2 that are Representation id of thetexture file (texture2 file) and the depth file (depth2 file) asassociationID with each other is described.

Further, in the representation element corresponding to the quality file(quality3 file), vq3 is described as Representation id, and“quality3.mp4” is described as BaseURL.

Further, quality information stored in the quality file (quality3 file)is quality information of a 3D image to be reproduced using the texturefile (texture1 file), the depth file (depth1 file), and the occlusionfile (occlusion file) and another 3D image to be reproduced using thetexture file (texture1 file), the depth file (depth2 file), and theocclusion file (occlusion file).

Accordingly, in the representation element corresponding to the qualityfile (quality3 file) group, vt1, vd1, vd2, and vo1 that areRepresentation id of the texture file (texture1 file), the depth file(depth1 file), the depth file (depth2 file), and the occlusion file(occlusion file) are described as associationID.

Further, in the example of FIG. 18, a track corresponding to the level 1has stored therein quality information of a 3D image to be reproducedusing the texture file (texture1 file), the depth file (depth1 file),and the occlusion file (occlusion file). A track corresponding to thelevel 2 has stored therein quality information of a 3D image to bereproduced using the texture file (texture1 file), the depth file(depth2 file), and the occlusion file (occlusion file)).

Accordingly, in the representation element corresponding to the qualityfile (quality3 file) group, <SubRepresentation level=“1”associationID=“vt1 vd1 vo1”> that associates the level 1 and vt1, vd1,and vo1 that are Representation id of the texture file (texture1 file),the depth file (depth1 file), and the occlusion file (occlusion file) isdescribed as associationID. Further, <SubRepresentation level=“2”associationID=“vt1 vd2 vo1”> that associates the level 2 and vt1, vd2,and vo1 that are Representation id of the texture file (texture1 file),the depth file (depth2 file), and the occlusion file (occlusion file) isdescribed as associationID.

It is to be noted that, in the example of FIG. 18, although bandwidth isnot described in the adaptation set element for a quality file, it mayotherwise be described.

Further, though not depicted, the file generation process in the thirdembodiment is same as the file reproduction process of FIG. 12 exceptthat the quality information acquired at step S11 is quality informationof a texture image or a 3D image reproduced in the reproduction patternand that an encoded stream of metadata including quality information isdivided not for each kind of a depth-related image but for each kind ofa reproduction pattern at step S14.

(Description of First Example of Process of Streaming Reproduction Unit)

FIG. 19 is a flow chart illustrating a first example of the reproductionprocess of the streaming reproduction unit 100 in the third embodiment.It is to be noted that, in the reproduction process of FIG. 19, thestreaming reproduction unit 100 performs reproduction of a 3D image inwhich a texture image and a depth image are used.

Processes at steps S91 to S95 of FIG. 19 are similar to the processes atsteps S31 to S35 of FIG. 14, respectively, and therefore, description ofthem is omitted. Processes at steps S94 to S103 are performed in a unitof a segment.

At step S96, the MPD processing unit 102 creates a list of reproductionpatterns for reproducing a texture image to be acquired.

For example, it is assumed that the maximum acceptable bit rate of adepth image in the MPD processing unit 102 is 20 percent of a networkbandwidth. Then, in the case where the texture image to be acquired is atexture image of the texture file (texture1 file) and the bit rate ofdepth images of the depth file (depth1 file) and the depth file (depth2file) is lower than the maximum acceptable bit rate, the MPD processingunit 102 creates a list in which the reproduction patterns 3 and 4 areregistered.

On the other hand, in the case where the texture image to be acquired isa texture image of the texture file (texture2 file) and the bit rate ofa depth image of the depth file (depth2 file) is lower than the maximumacceptable bit rate, the MPD processing unit 102 creates a list in whichthe reproduction pattern 7 is registered. Then, the MPD processing unit102 supplies acquisition information of quality information of thereproduction patterns registered in the list to the quality informationacquisition unit 103.

It is to be noted that, in the case where the bit rates of all depthimages to be reproduced together with a texture image to be acquired isequal to or higher than the maximum acceptable bit rate, nothing isregistered into the list of acquisition candidates for a depth image andonly an encoded stream of the texture image is acquired, decoded, anddisplayed, whereafter the process advances to step S103.

At step S97, the quality information acquisition unit 103 requests, onthe basis of the acquisition information supplied from the MPDprocessing unit 102, the Web server 12 for an encoded stream of metadataincluding quality information of a predetermined reproduction pattern toacquire the encoded stream. The quality information acquisition unit 103supplies the acquired encoded stream to the decoding unit 104.

At step S98, the decoding unit 104 decodes the encoded stream of qualityinformation of the predetermined reproduction pattern supplied from thequality information acquisition unit 103 and generates metadataincluding the quality information of the predetermined reproductionpattern. The decoding unit 104 supplies the quality information of thepredetermined reproduction pattern to the MPD processing unit 102.

At step S99, the MPD processing unit 102 determines, on the basis of thequality information supplied from the decoding unit 104, a depth imageto be acquired from among depth images registered in the list and to beused for reproduction of a reproduction pattern.

For example, the MPD processing unit 102 determines a depth image to beused in reproduction in a reproduction pattern in which the qualityrepresented by the quality information is best, a reproduction patternwhose quality represented by the quality information is nearest to thequality of a reproduction pattern of the immediately preceding segment(or sub segment), or a reproduction pattern in which the qualityrepresented by the quality information is acceptable quality and besidesthe bit rate is lowest as a depth image to be acquired.

In the case where a depth image to be used in reproduction in areproduction pattern whose quality represented by the qualityinformation is nearest to the quality of a reproduction pattern of theimmediately preceding segment (or sub segment) is determined as a depthimage to be acquired, the sense of incongruity of the appearance of thereproduced 3D image can be reduced. The MPD processing unit 102 suppliesacquisition information of the depth image to be acquired to the imageacquisition unit 105.

Processes at steps S100 to S103 are similar to the processes at stepsS40 to S43 of FIG. 14, respectively, and therefore, description of themis omitted.

It is to be noted that, though not depicted, the reproduction processwhen the streaming reproduction unit 100 performs reproduction of a 3Dimage using a texture image, a depth image, and an occlusion image issimilar to the reproduction process of FIG. 19 except the followingpoint.

In particular, candidates for a reproduction pattern selected at stepS93 of FIG. 19 are the reproduction patterns 5 and 6. Further, the depthimage in the processes at steps S99 to S102 is replaced by a depth imageand an occlusion image. It is to be noted that, in this case, themaximum acceptable bit rates of the texture image, the depth image, andthe occlusion image are, for example, 70 percent, 15 percent, and 15percent of the network bandwidth, respectively.

(Description of Second Example of Process of Streaming ReproductionUnit)

FIG. 20 is a flow chart illustrating a second example of thereproduction process of the streaming reproduction unit 100 in the thirdembodiment. It is to be noted that, in the reproduction process of FIG.20, the streaming reproduction unit 100 performs reproduction of a 3Dimage using a texture image and a depth image.

The reproduction process of FIG. 20 is different from the reproductionprocess of FIG. 19 principally in that the ratio of the maximumacceptable bit rates for a texture image and a depth image to thenetwork bandwidth is not determined.

Processes at steps S121 to S124 of FIG. 20 are similar to the processesat steps S91 to S94 of FIG. 19, respectively, and therefore, descriptionof them is omitted. Processes at steps S124 to S132 are performed in aunit of a segment.

At step S125, the MPD processing unit 102 creates a list of reproductionpatterns on the basis of candidates for a reproduction pattern, thenetwork bandwidth of the Internet 13 supplied from the measurement unit108, and the bit rates for a texture image and a depth image.

In particular, the MPD processing unit 102 creates a list in which, fromamong the reproduction patterns 3, 4, and 7, reproduction patterns inwhich the sum of the bit rates for a texture image and a depth image tobe used for reproduction does not exceed the network bandwidth areregistered.

It is to be noted that lower limits for the bit rates for a textureimage and a depth image may be determined in advance such that, fromamong the reproduction patterns registered in the list, any reproductionpattern in which at least one of the bit rates for a texture image and adepth image to be used for reproduction is lower than the lower limittherefor is excluded.

Further, in the case where all of the sums of the bit rates for atexture image and a depth image that are used for reproduction in thereproduction patterns 3, 4, and 7 exceed the network bandwidth, nothingis registered to the list of reproduction patterns. Then, only anencoded stream of a texture image of a maximum bit rate that does notexceed the network bandwidth is acquired, decoded, and displayed, andthe process advances to step S132.

Processes at steps S126 and S127 are similar to the processes at stepsS97 and S98 of FIG. 19, respectively, and therefore, description of themis omitted.

At step S128, the MPD processing unit 102 determines a combination of atexture image and a depth image to be acquired similarly as in theprocess at S69 of FIG. 15 from among the combinations of a texture imageand a depth image to be used for reproduction in the reproductionpatterns registered in the list on the basis of the quality information.

Processes at steps S129 to S132 are similar to the processes at stepsS100 to S103 of FIG. 19, respectively, and therefore, description ofthem is omitted.

It is to be noted that, though not depicted, the second example of thereproduction process when the streaming reproduction unit 100 performsreproduction of a 3D image using a texture image, a depth image, and anocclusion image is similar to the reproduction process of FIG. 20 exceptthe following point.

In particular, the candidates for a reproduction pattern selected atstep S123 of FIG. 20 are the reproduction patterns 5 and 6. Further, thedepth image in the processes at steps S128 to S131 includes both a depthimage and an occlusion image.

As described above, the file generation apparatus 11 in the thirdembodiment divides quality information of a pattern to be reproducedinto quality information for each kind of a reproduction pattern anddisposes the divisional quality information into different qualityfiles. Accordingly, the number of quality files can be reduced incomparison with that in an alternative case in which quality informationis disposed in quality files different among different patterns to bereproduced. Therefore, it can be said that quality information can bestored efficiently. Further, the processing amount relating toacquisition of quality information of the video reproduction terminal 14can be reduced.

Further, the video reproduction terminal 14 can acquire qualityinformation from a quality file in which only quality information of areproduction pattern of a kind to be made a candidate is stored.Accordingly, the acquisition efficiency of quality information of areproduction pattern of a candidate can be improved in comparison withthat in an alternative case in which quality information is acquiredfrom quality files in which the quality information of all reproductionpatterns is stored.

Fourth Embodiment (Example of Segment File)

The configuration of a fourth embodiment of the information processingsystem to which the present disclosure is applied is same as theconfiguration of the third embodiment principally except that an encodedstream of metadata including quality information is divided not for eachkind of reproduction pattern but for each texture image used forreproduction and each of the divisions is disposed into a different oneof quality files. Accordingly, in the following description, descriptionother than description of a quality file is omitted suitably.

FIG. 21 depicts an example of a segment file generated by the segmentfile generation unit 83 of the fourth embodiment of the informationprocessing system to which the present disclosure is applied.

The segment file of FIG. 21 is same as the segment file of FIG. 5 excepta quality file.

As depicted in FIG. 21, the segment file generation unit 83 divides anencoded stream of metadata including quality information of reproductionpatterns 1 to 7 supplied from the encoding unit 82 into two for eachtexture image to be reproduced in the reproduction patterns. Then, thesegment file generation unit 83 disposes the divisions of the encodedstream of metadata in a unit of a segment into the different qualityfiles to generate quality files.

In particular, the segment file generation unit 83 generates a qualityfile (quality1 file) in which encoded streams of quality information ofthe reproduction patterns 1 to 5 to be reproduced using a texture file(texture1 file) are disposed in a unit of a segment. In the quality file(quality1 file), the respective encoded streams are disposed indifferent tracks (quality track(texture1), qualitytrack(texture1+depth1), quality track(texture1+depth2), qualitytrack(texture1+depth1+occlusion), and qualitytrack(texture1+depth2+occlusion)).

Further, the segment file generation unit 83 generates a quality file(quality2 file) in which encoded streams of quality information of thereproduction patterns 6 and 7 to be reproduced using a texture file(texture2 file) are disposed in a unit of a segment. In the quality file(quality2 file), the respective encoded streams are disposed indifferent tracks (quality track(texture2) and qualitytrack(texture2+depth2)).

As described above, the segment file generation unit 83 files encodedstreams of quality information of a reproduction pattern separately foreach texture image. Accordingly, by acquiring quality information of aquality file of a texture image to be acquired, the video reproductionterminal 14 can easily acquire quality information of patterns to bereproduced for performing reproduction using the texture image.

Further, though not depicted, the MPD file in the fourth embodiment issimilar to the MPD file of FIG. 18 except the following point. Inparticular, in the MPD file in the fourth embodiment, the number ofrepresentation elements included in the adaptation set element for aquality file is two.

associationID a representation element of the first quality file(quality1 file) group has is vt1, vt2, vd1, vd2, and vo1. Further, arepresentation element of the quality file (quality1 file) includes fivesub representation elements having vt1 as the level 1 and associationID,vt1 and vd1 as the level 2 and associationID, vt1 and vd2 as the level 3and associationID, vt1, vd1, and vo1 as the level 4 and associationID,and vt1, vd2, and vo1 as the level 5 and associationID.

associationID the representation element of the second quality file(quality2 file) has is vt2 and vd2. Further, the representation elementof the quality file (quality1 file) includes two sub representationelements individually having the level 1 and vt2 as associationID andthe level 2 and vt2 and vd2 as associationID.

Further, though not depicted, the file generation process in the fourthembodiment is same as the file generation process in the thirdembodiment except that an encoded stream of metadata including qualityinformation is divided not for each kind of reproduction pattern but foreach texture image.

Further, though not depicted, the reproduction process in the fourthembodiment is same as the reproduction process of FIG. 19 or FIG. 20.

As described above, the file generation apparatus 11 in the fourthembodiment divides quality information of a pattern to be reproduced foreach texture image and disposes the divisions of the quality informationinto individually different quality files. Accordingly, in comparisonwith an alternative case in which quality information is disposed inquality files different among different patterns to be reproduced.Therefore, it can be said that quality information of a pattern to bereproduced can be stored efficiently. Further, the processing amountrelating to acquisition of quality information of the video reproductionterminal 14 can be reduced.

Further, the video reproduction terminal 14 can acquire qualityinformation from a quality file in which only quality information of apattern to be reproduced in which reproduction is performed using atexture image of a reproduction target is stored. Accordingly, incomparison with an alternative case in which quality information isacquired from a quality file in which quality information of allreproduction patterns is stored, the acquisition efficiency of qualityinformation of a pattern to be reproduced in which reproduction isperformed using a texture image of a reproduction target can beimproved.

Fifth Embodiment (Example of Segment File)

The configuration of a fifth embodiment of the information processingsystem to which the present disclosure is applied is same as theconfiguration of the third configuration principally except that anencoded stream of metadata including quality information is divided notonly for each kind of reproduction pattern but also for each textureimage is disposed into a different one of quality files. In particular,the fifth embodiment is a combination of the third embodiment and thefourth embodiment. Accordingly, in the following description,description other than description of a quality file is omittedsuitably.

FIG. 22 depicts an example of a segment file generated by the segmentfile generation unit 83 of the fifth embodiment of the informationprocessing system to which the present disclosure is applied.

The segment file of FIG. 22 is same as the segment file of FIG. 5 excepta quality file.

As depicted in FIG. 22, the segment file generation unit 83 dividesencoded streams of metadata including quality information ofreproduction patterns 1 to 7 supplied from the encoding unit 82 intofive for each kind of reproduction pattern and for each texture image tobe used for reproduction in each reproduction pattern. Then, the segmentfile generation unit 83 disposes the divisions of the encoded streams ina unit of a segment into the different quality files to generate qualityfiles.

In particular, the segment file generation unit 83 generates a qualityfile (quality1 file) in which an encoded stream of quality informationof the reproduction pattern 1 to be reproduced using a texture file(texture1 file) is disposed in a unit of a segment.

Further, the segment file generation unit 83 generates a quality file(quality2 file) in which encoded streams of quality information of thereproduction patterns 3 and 4 to be reproduced using only the texturefile (texture1 file) and a depth file are disposed in a unit of asegment. In the quality file (quality2 file), the respective encodedstreams are disposed in different tracks (quality track(texture1+depth1)and quality track(texture1+depth2)).

Furthermore, the segment file generation unit 83 generates a qualityfile (quality3 file) in which encoded streams of quality information ofthe reproduction patterns 5 and 6 to be reproduced using only thetexture file (texture1 file), the depth file, and the occlusion file aredisposed in a unit of a segment. In the quality file (quality3 file),the respective encoded streams are disposed on different tracks (qualitytrack(texture1+depth1+occlusion) and qualitytrack(texture1+depth2+occlusion)).

Further, the segment file generation unit 83 generates a quality file(quality4 file) in which an encoded stream of quality information of thereproduction pattern 2 to be reproduced using only the texture file(texture2 file) is disposed in a unit of a segment.

Furthermore, the segment file generation unit 83 generates a qualityfile (quality5 file) in which an encoded stream of quality informationof the reproduction pattern 7 to be reproduced using only the texturefile (texture2 file) and the depth file is disposed in a unit of asegment.

As described above, the segment file generation unit 83 files an encodedstream of quality information of a reproduction pattern separately foreach kind of reproduction pattern and for each texture image.Accordingly, from a quality file of a pattern to be reproduced in whichreproduction is performed using a texture image of a reproductiontarget, which is a kind of reproduction patterns that become candidates,quality information of a plurality of patterns that become candidatesand are to be reproduced in which reproduction is performed using atexture image of the quality file.

Further, though not depicted, the MPD file in the fifth embodiment issimilar to the MPD file of FIG. 18 except the following point. Inparticular, in the MPD file in the fifth embodiment, the number ofrepresentation elements included in an adaptation set element for aquality file is five.

associationID a representation element of the first quality file(quality1 file) has is vt1, and associationID a representation elementof the fourth quality file (quality4 file) has is vt2. associationID arepresentation element of the fifth quality file (quality5 file) has isvt2 and vd2.

associationID a representation element of the second quality file(quality2 file) has is vt1, vd1, and vd2. Meanwhile, a representationelement of the quality file (quality1 file) includes sub representationelements individually having the level 1 and vt1 and vd1 asassociationID and the level 2 and vt1 and vd2 as associationID.

associationID a representation element of the third quality file(quality2 file) has is vt1, vd1, vd2, and vo1. Meanwhile, arepresentation element of the quality file (quality1 file) includes subrepresentation elements individually having the level 1 and vt1, vd1,and vo1 as associationID and the level 2 and vt1, vd2, and vo1 asassociationID.

Further, though not depicted, the file generation process in the fifthembodiment is same as the file generation process in the thirdembodiment except that an encoded stream of metadata including qualityinformation is divided not only for each kind of a reproduction patternbut also for each texture image. Further, though not depicted, thereproduction process in the fifth embodiment is same as the reproductionprocess of FIG. 19 or FIG. 20.

As described above, the file generation apparatus 11 in the fifthembodiment divides quality information of a pattern to be reproduced foreach kind of reproduction pattern and for each texture image anddisposes the divisions of the quality information into different qualityfiles. Accordingly, in comparison with an alternative case in whichquality information is disposed in quality files different amongdifferent patterns to be reproduced, the number of quality files can bereduced. Therefore, it can be said that quality information of a patternto be reproduced can be stored efficiently. Further, the processingamount relating to acquisition of quality information by the videoreproduction terminal 14 can be reduced.

Further, the video reproduction terminal 14 can acquire qualityinformation from a quality file in which only quality information of apattern to be reproduced, which is a kind of reproduction pattern thatbecomes a candidate and in which reproduction is performed using atexture image of a reproduction target. Accordingly, in comparison withan alternative case in which quality information is acquired from aquality file in which quality information of all reproduction patternsis stored, the acquisition efficiency of quality information of apattern to be reproduced in which reproduction is performed using atexture image of a reproduction target can be improved.

Sixth Embodiment (Example of Segment File)

The configuration of a sixth embodiment of the information processingsystem to which the present disclosure is applied is same as theconfiguration of the information processing system 10 of FIG. 1principally except that encoded streams of metadata including qualityinformation are disposed collectively into a single quality file.Accordingly, in the following description, description other thandescription of a quality file is omitted suitably.

FIG. 23 is a view depicting an example of a segment file generated bythe segment file generation unit 83 in the sixth embodiment of theinformation processing system to which the present disclosure isapplied.

The segment file of FIG. 23 is same as the segment file of FIG. 5 excepta quality file.

As depicted in FIG. 23, the segment file generation unit 83 generatesone quality file (quality1 file) by disposing encoded streams ofmetadata including quality information of depth images of 2 Mbps and 1Mbps and an encoded stream of metadata including quality information ofan occlusion image of 1 Mbps into one quality file (quality1 file) in aunit of a segment.

In the quality file (quality1 file), different tracks (qualitytrack(depth1), quality track(depth2), and quality track(Occlusion1)) areallocated to the different encoded streams.

(Example of Description of MPD File)

FIG. 24 is a view depicting an example of a description of an MPD filein the sixth embodiment.

The configuration of the MPD file of FIG. 24 is same as theconfiguration of FIG. 10 except an adaptation set (AdaptationSet)element for a quality file.

In the MPD file of FIG. 24, a quality file (quality1 file) group isgrouped by one adaptation set element.

In the adaptation set element for a quality file, a representationelement corresponding to the quality file (quality1 file) is described.In the representation element corresponding to the quality file(quality1 file), vq1 is described as Representation id, and“quality1.mp4” is described as BaseURL.

Further, the quality information stored in the quality file (quality1file) is quality information of the depth file (depth1 file), the depthfile (depth2 file), and the occlusion file (occlusion1 file).Accordingly, in the representation element corresponding to the qualityfile (quality1 file), vd1, vd2, and vo1 that are Representation id ofthe depth file (depth1 file) group, the depth file (depth2 file), andthe occlusion file (occlusion1 file) group are described asassociationID.

Further, in the example of FIG. 24, tracks corresponding to the level 1to the level 3 have stored individually therein quality information ofthe depth file (depth1 file), the depth file (depth2 file), and theocclusion file (occlusion1 file).

Accordingly, in the representation element corresponding to the qualityfile (quality1 file) group, <SubRepresentation level=“1”associationID=“vd1”> that associates the level 1 and vd1 that isRepresentation id of the depth file (depth1 file) as associationID witheach other is described. <SubRepresentation level=“2”associationID=“vd2”> that associates the level 2 and vd2 that isRepresentation id of the depth file (depth2 file) as associationID witheach other is described. <SubRepresentation level=“3”associationID=“vo1”> that associates the level 3 and vo1 that isRepresentation id of the occlusion file (occlusion1 file) asassociationID with each other is described.

It is to be noted that, although, in the example of FIG. 24, bandwidthis not described in the adaptation set element for a quality file, itmay otherwise be described.

Further, though not depicted, the file generation process in the sixthembodiment is same as the file generation process of FIG. 12 except thatthe process at step S14 is not performed and encoded streams of metadataincluding quality information are disposed into one quality file in aunit of a segment at step S15.

Further, though not depicted, the reproduction process in the sixthembodiment is same as the reproduction process in FIG. 14 or FIG. 15.

It is to be noted that, while, in the first and sixth embodiments, acombination of images in a pattern to be reproduced is described in anMPD file, it may not be described. In this case, candidates for areproduction pattern are selected from among reproduction patterns inwhich all combinations of images that can be combined are used.

Further, encoded streams of quality information may be disposed inquality files different for the individual encoded streams.

Seventh Embodiment (Example of Configuration of Segment File)

The configuration of a seventh embodiment of the information processingsystem to which the present disclosure is applied is same as theconfiguration of the information processing system 10 of FIG. 1principally except that encoded streams of depth occlusion images andencoded streams of quality information of depth occlusion images aredivided for each kind of depth occlusion image and disposed in differentsegment files and that the encoded streams of quality information withinthe same segment file are disposed on the same track. Accordingly, inthe following description, description other than description of asegment file for each kind of depth occlusion image is omitted suitably.

FIG. 25 is a view depicting an example of a segment file generated bythe segment file generation unit 83 of the seventh embodiment of theinformation processing system to which the present disclosure isapplied.

The texture file of FIG. 25 is same as the texture file of FIG. 5. Asdepicted in FIG. 25, the segment file generation unit 83 divides encodedstreams of quality information of depth images of 2 Mbps and 1 Mbps andocclusion image of 1 Mbps and encoded streams of depth images of 2 Mbpsand 1 Mbps and occlusion image of 1 Mbps into two for each kind of depthocclusion image.

Then, the segment file generation unit 83 generates, in a unit of asegment, a segment file in which encoded streams of divisions of depthimages of 2 Mbps and 1 Mbps and an encoded stream of quality informationof divisions of a depth image of 1 Mbps are disposed.

In particular, the segment file generation unit 83 generates a depthfile (depth1 file) in which encoded streams of depth images of 2 Mbpsand 1 Mbps and an encoded stream of quality information of depth imagesof 2 Mbps and 1 Mbps are disposed.

In the depth file (depth1 file), encoded streams of quality informationof depth images of 2 Mbps and 1 Mbps are disposed on the same track(quality1 track), and an encoded stream of a depth image of 2 Mbps andan encoded stream of a depth image of 1 Mbps are disposed on tracksseparate from each other (depth1 track, depth2 track).

Accordingly, in the track (quality1 track) of the encoded streams ofquality information of the depth images of 2 Mbps and 1 Mbps, both ofthe encoded streams of the quality information of the depth images of 2Mbps and 1 Mbps are collectively sampled.

Further, the segment file generation unit 83 generates an occlusion file(occlusion1 file) in which an encoded stream of occlusion image of 1Mbps and an encoded stream of quality information of occlusion image of1 Mbps are disposed.

As described above, in the seventh embodiment, encoded streams ofquality information disposed in a segment file of the same depthocclusion image are disposed on the same track. Accordingly, incomparison with an alternative case in which encoded streams of qualityinformation disposed on a segment file of the same depth occlusion imageare disposed on tracks different from each other for different encodedstreams, the track number in a segment file of depth occlusion imagescan be reduced. As a result, the size of segment files of depthocclusion images can be reduced. Further, the load on the videoreproduction terminal 14 can be reduced.

(Example of Configuration of Sample)

FIG. 26 is a view depicting an example of a configuration of a sample ofa track (quality1 track) of FIG. 25.

As depicted in FIG. 26, the sample of the track (quality1 track) isdivided into two subsamples, into each of which encoded streams of depthimages of 2 Mbps and 1 Mbps are disposed in a divided form.

In the example of FIG. 26, in the first subsample of the ith (i=1, 2, .. . , n) sample, an encoded stream (Depth1 Quality i) of a depth imageof 2 Mbps is disposed, and in the second subsample, an encoded stream(Depth2 Quality i) of a depth image of 1 Mbps is disposed. Details ofthe subsample are described in ISO/IEC 23001-10.

(Example of Configuration of Moov Box of Depth File)

FIG. 27 is a view depicting an example of a configuration of a moov box(movie box) of the depth file (depth1 file).

As depicted in FIG. 27, in the moov box of the depth file (depth1 file),a trak box is disposed for each track. In the trak box, a tkhd box(track header box) in which a track ID (track_ID) that is an ID uniqueto the track is described is disposed.

In the example of FIG. 27, the track ID of the track (depth1 track) inwhich an encoded stream of a depth image of 2 Mbps is disposed is 1, andthe track ID of the track (depth2 track) in which an encoded stream of adepth image of 1 Mbps is disposed is 2. Further, the track ID of thetrack (quality1 track) in which an encoded stream of quality informationof the depth images of 2 Mbps and 1 Mbps is disposed is 3.

Further, in the trak box, a tref box (track reference) in which thetrack ID of a different track having a relation to the own track isdescribed can be disposed. In particular, the track (quality1 track) isa track on which an encoded stream of quality information of the encodedstreams of the depth image disposed on the track (depth1 track) and thetrack (depth2 track) is stored. Accordingly, on the track (quality1track), a tref box in which 1 and 2 that are the track IDs of the track(depth1 track) and the track (depth2 track) are described is disposed.

Consequently, the video reproduction terminal 14 can recognize that thetrack (quality1 track) has accommodated therein an encoded stream ofquality information of encoded streams of depth images disposed on thetrack (depth1 track) and the track (depth2 track).

However, the video reproduction terminal 14 cannot recognize whichsubsample of the track (quality1 track) stores the encoded stream of thequality information of an encoded stream of the depth image accommodatedon the track (depth1 track) or the track (depth2 track).

Accordingly, in the seventh embodiment, a corresponding relationship(hereinafter referred to as subsample track correspondence relationship)between a subsample and a track ID (track specification information) forspecifying the track of a depth occlusion image corresponding to theencoded stream of the quality information disposed in the subsample isdescribed. Consequently, the video reproduction terminal 14 canrecognize the quality information of an encoded stream of a depthocclusion image stored in each subsample. As a result, the videoreproduction terminal 14 can acquire an encoded stream of qualityinformation of each depth occlusion image from the track (quality1track).

Although, in the seventh embodiment, a subsample track correspondencerelationship is described in QualityMetricsConfigurationBox,QualityMetricsSampleEntry, SubsampleInformationBox, orSubsampleReferenceBox, it may otherwise be described in any box otherthan those described above.

(Example of Description of QualityMetricsConfigurationBox)

FIG. 28 is a view depicting an example of a description ofQualityMetricsConfigurationBox in the case where a subsample trackcorrespondence relationship is described inQualityMetricsConfigurationBox disposed in the trak box of the track(quality1 track).

In QualityMetricsConfigurationBox of FIG. 28, field_size_bytes andmetric_count are described, and the number of metric_code equal tometric_count is described. field_size_bytes, metric_count, andmetric_count are similar to those in the case of FIG. 6 except that thequality file is replaced by the track (quality1 track).

In the seventh embodiment, in the quality file, encoded streams of twokinds of quality information of the track (depth1 track) and the track(depth2 track) are stored in a sample of the track (quality1 track).Accordingly, metric_count is 2.

Meanwhile, in QualityMetricsConfigurationBox, 1 indicating that asubsample track correspondence relationship can be described is set as aflag.

In the case where flag is 1, in QualityMetricsConfigurationBox,referenced_track_in_file_flag indicative of whether a track to bereferred to exists in the depth file (depth1 file) including the track(quality1 track) is described for each metric_code.

In the case where referenced_track_in_file_flag of each metric_code is1, in QualityMetricsConfigurationBox, reference_track_id_num of asubsample corresponding to metric_code and the number of track_id equalto reference_track_id_num are described.

reference_track_id_num is the number of tracks in the depth file (depth1file) to be referred to. track_id is a track ID of a track in the depthfile (depth1 file) to be referred to.

In the seventh embodiment, the first subsample in a sample of the track(quality1 track) corresponds to the track (quality1 track) in the depthfile (depth1 file), and the second subsample corresponds to the track(depth2 track). Accordingly, referenced_track_in_file_flag ofmetric_code corresponding to each subsample is set to 1 that indicatesthat a track to be referred to exists in the depth file (depth1 file)including the track (quality1 track).

Further, reference_track_id_num of metric_code corresponding to eachsubsample is 1. Furthermore, track_id of metric_code corresponding tothe first subsample is 1 that is the track ID (track specificationinformation) that specifies the track (depth1 track) of a depth imagecorresponding to an encoded stream of quality information disposed inthe subsample. Meanwhile, track_id of metric_code corresponding to thesecond subsample is 2 that is the track ID of the track (depth2 track)of the depth image corresponding to the encoded stream of qualityinformation disposed in the subsample.

In this manner, in QualityMetricsConfigurationBox, track IDs of depthimages corresponding to subsamples are described in an order in whichthe subsamples are disposed in a sample thereby to describe a subsampletrack correspondence relationship.

(Example of Description of QualityMetricsSampleEntry)

FIG. 29 is a view depicting an example of a description ofQualityMetricsSampleEntry in the case where a subsample trackcorrespondence relationship is described in QualityMetricsSampleEntrydisposed in the trak box of the track (quality1 track).

In QualityMetricsSampleEntry of FIG. 29, QualityMetricsReferenceBox isdisposed. In QualityMetricsReferenceBox, metric_count is describedsimilarly as in QualityMetricsConfigurationBox of FIG. 28. Further,referenced_track_in_file_flag is described in order of metric_code ofQualityMetricsConfigurationBox, and in the case wherereferenced_track_in_file_flag of each metric_code is 1,reference_track_id_num of this metric_code and the number of track_idequal to reference_track_id_num are described.

(Example of Description of SubsampleInformationBox)

FIG. 30 is a view depicting an example of a description ofSubsampleInformationBox in the case where a subsample trackcorrespondence relationship is described in SubsampleInformationBox isdisposed in the trak box of the track (quality1 track).

SubsampleInformationBox is a box that describes information relating toa subsample. The trak box can have a plurality ofSubSampleInformationBox having values of flags different from eachother. In SubsampleInformationBox of FIG. 30, 2 indicating that asubsample track correspondence relationship can be described is set asversion.

In the case where version is greater than 1, in SubsampleInformationBox,track_reference_is_exist_flag and referenced_track_in_file_flag aredescribed for each subsample. track_reference_is_exist_flag is a flagindicative of whether there is the necessity for expansion to make itpossible to describe a subsample track correspondence relationship. Thismakes it possible to prevent, when a value equal to or greater than 3 isset as version, expansion that makes it possible to describe a subsampletrack correspondence relationship from being performed uselessly.

In the seventh embodiment, since it is necessary to expand to make itpossible to describe a subsample track correspondence relationship,track_reference_is_exist_flag is set to 1 indicating that it isnecessary to expand to make it possible to describe a subsample trackcorrespondence relationship.

In the case where both track_reference_is_exist_flag andreferenced_track_in_file_flag of each subsample are 1, inSubsampleInformationBox, reference_track_id_num of the subsample and thenumber of track_id equal to reference_track_id_num are described.

(Example of Description of SubsampleReferenceBox)

FIG. 31 is a view depicting an example of a description ofSubsampleReferenceBox in the case where a subsample track correspondencerelationship is described in SubsampleReferenceBox is disposed in thetrak box of the track (quality1 track).

In SubsampleReferenceBox of FIG. 31, referenced_track_in_file_flag isdescribed for each subsample, and in the case wherereferenced_track_in_file_flag of each subsample is 1, thereference_track_id_num of the subsample and the number of track_id equalto reference_track_id_num are described.

(Example of Description of MPD File)

FIG. 32 is a view depicting an example of a description of an MPD filein the seventh embodiment.

The configuration of the MPD file of FIG. 32 is same as theconfiguration of the MPD file of FIG. 10 except the configuration of theadaptation set element for a depth file and the adaptation set elementfor an occlusion file and except that the adaptation set element for aquality file is not provided.

In the adaptation set element for a depth file of FIG. 32, arepresentation element corresponding to a depth file (depth1 file) groupis described.

In the representation element corresponding to the depth file (depth1file), vd1 is described as Representation id, and “depth1.mp4” isdescribed as BaseURL. It is to be noted that, although bandwidth is notdescribed in the example of FIG. 32, bandwidth may otherwise bedescribed.

Further, the texture file to be reproduced together with the depth file(depth1 file) is the texture file (texture1 file) or the texture file(texture2 file). Accordingly, in the representation elementcorresponding to the depth file (depth1 file), vt1 and vt2 that areRepresentation id of the texture file (texture1 file) and the texturefile (texture2 file) are described.

Furthermore, in the example of FIG. 32, the track (depth1 track) and thetrack (depth2 track) are associated with the levels 1 and 2,respectively, by a leva box.

Accordingly, in the representation element corresponding to the depthfile (depth1 file) group of FIG. 32, <SubRepresentation level=“1”associationID=“vt1”> that associates the level 1 and vt1 asassociationID that is Representation id of the texture file (texture1file) to be reproduced together with the track (depth1 track) isdescribed.

Similarly, <SubRepresentation level=“2” associationID=“vt1 vt2”> thatassociates the level 2 and vt1 and vt2 as associationID that areRepresentation id of the texture file (texture1 file) and the texturefile (texture2 file) to be reproduced together with the track (depth2track).

In the representation element corresponding to the occlusion file(occlusion1 file) group, vo1 is described as Representation id and“occlusion1.mp4” is described as BaseURL. It is to be noted that,although bandwidth is not described in the example of FIG. 32, bandwidthmay otherwise be described.

Further, the depth file to be reproduced together with the occlusionfile (occlusion1 file) is the depth file (depth1 file). Accordingly, inthe representation element corresponding to the occlusion file(occlusion1 file), vd1 that is Representation id of the depth file(depth1 file) is described as associationID.

It is to be noted that, while, in the example of FIG. 32, thecombination of images to be used in a pattern to be reproduced is notdescribed in the MPD file, it may otherwise be described.

Further, though not depicted, the file generation process in the seventhembodiment is same as the file generation process of FIG. 12 exceptthat, in the depth file and the occlusion file generated at step S13,corresponding quality information is disposed and the processes at stepsS14 and S15 are not performed.

Furthermore, though not depicted, the reproduction process in theseventh embodiment is same as the reproduction process of FIG. 14 orFIG. 15 except that an encoded stream is acquired referring also to asubsample track correspondence relationship.

In this manner, the file generation apparatus 11 according to theseventh embodiment collectively disposes a plurality of encoded streamsof quality information disposed in the segment file of depth occlusionimages into one track. Accordingly, in comparison with an alternativecase in which encoded streams of different kinds of quality informationare disposed on different tracks, the number of tracks that configure asegment file of depth occlusion images can be reduced. In other words,quality information of depth occlusion images can be stored efficiently.Consequently, the size of the segment file of depth occlusion images isreduced. As a result, the transmission amount when the file generationapparatus 11 uploads a segment file of depth occlusion images isreduced.

It is to be noted that, while, in the seventh embodiment, the qualityinformation is quality information of occlusion images, the qualityinformation may otherwise be quality information of a 3D image that isreproduced using a texture image or a texture image and a depthocclusion image.

In this case, for example, a track of a texture image and a track ofquality information of the texture image are disposed in a texture file,and a track of a depth image and a track of quality information of a 3Dimage reproduced using a texture image and the depth image are disposedin a depth file. Further, a track of an occlusion image and a track ofquality information of a 3D image reproduced using a texture image, adepth image, and an occlusion image are disposed in an occlusion file.

Eighth Embodiment (Example of Configuration of Segment File)

The configuration of an eighth embodiment of the information processingsystem to which the present disclosure is applied is same as theconfiguration of the information processing system 10 of FIG. 1principally except that encoded streams of quality information in a samequality file are disposed on the same track. Accordingly, in thefollowing description, description other than description of a qualityfile (quality1 file) is omitted suitably.

FIG. 33 is a view depicting an example of a segment file generated bythe segment file generation unit 83 of the eighth embodiment of theinformation processing system to which the present disclosure isapplied.

The segment file of FIG. 33 is same as the segment file of FIG. 5 excepta quality file (quality1 file) of depth images.

As depicted in FIG. 33, the segment file generation unit 83 generates aquality file (quality1 file) in which encoded streams of qualityinformation of depth images of 2 Mbps and 1 Mbps are collectivelydisposed on one track (quality1 track). On the track (quality1 track),both of the encoded streams of quality information of depth images of 2Mbps and 1 Mbps are sampled collectively.

The MPD file in the eighth embodiment is same as the MPD file of FIG.10. In particular, in the MPD file in the eighth embodiment, acorrespondence relationship between each level and associationID thatspecifies a depth occlusion image is described in a representationelement corresponding to the quality file (quality1 file).

(First Example of Description of Leva Box)

FIG. 34 is a view depicting a first example of a description of the levabox of the quality file (quality file).

As depicted in FIG. 34, in the leva box of the quality file (qualityfile), level_count indicative of the number of levels a subrepresentation element, which corresponds to a quality file (qualityfile) described in the MPD file, has is described.

Further, in the leva box of the quality file (quality file), the number,equal to level_count, of track_id, assignment_type and so forth of eachlevel are described in order beginning with the level 1. assignment_typeis a type of something associated with the level.

In the leva box of FIG. 34, 5 can be set as assignment_type. In theexample of FIG. 34, 5 as assignment_type represents that the typeassociated with the level is metric_code described inQualityMetricsConfigurationBox. In particular, in the case whereassignment_type is 5, the level i (i=1, 2) and the ith metric_code fromthe top describe in QualityMetricsConfigurationBox are associated witheach other.

(Description of First Example of Level and Subsample Associated by LevaBox)

FIG. 35 is a view illustrating a first example of the level and thesubsample in the quality file (quality1 file) associated with each otherby the leva box of FIG. 34.

In the MPD file of the eighth embodiment, the number of levels the subrepresentation element corresponding to the quality file (quality1 file)has is two. Accordingly, as depicted in FIG. 35, 2 is described aslevel_count in the leva box of the quality file (quality file)

Further, the track corresponding to the two levels is the track(quality1 track) whose track ID is 1. Accordingly, in the leva box ofthe quality file (quality file), 1 is described as track_id of the twolevels. Further, 5 is described as assignment_type of the two levels.

Accordingly, by the leva box of FIG. 35, the level i each subrepresentation element has can be associated with the ith metric_codefrom the top described in QualityMetricsConfigurationBox. Further, bythe description of QualityMetricsConfigurationBox, the ith metric_codefrom the top described in QualityMetricsConfigurationBox and the ithsubsample from the top can be associated with each other.

As described above, by describing 5 as assignment_type of the leva box,the level i described in the MPD file and the ith subsample can beassociated with each other. Accordingly, 5 as assignment_type can besaid as information that associates a level and a subsample with eachother.

(Second Example of Description of Leva Box)

FIG. 36 is a view depicting a second example of a description of theleva box of the quality file (quality file).

The leva box of the quality file (quality file) of FIG. 36 has aconfiguration same as that of FIG. 34 except a case in which 5 is set asassignment_type.

In the example of FIG. 36, 5 as assignment_type represents that the typeof something associated with a level is a subsample in which informationis described in a subs box (Sub-Sample Information Box). In particular,in the case where assignment_type is 5, the level i and the ithsubsample from the top in which information is described in the subsbox. Further, in the leva box of FIG. 36, in the case whereassignment_type is 5, flags of the subs box associated with the level iis described.

(Description of Second Example of Level and Subsample Associated by LevaBox)

FIG. 37 is a view illustrating a second example of a level and asubsample in the quality file (quality1 file) associated with each otherby the leva box of FIG. 36.

The leva box of FIG. 37 is same as the leva box of FIG. 35 except that 0is described as subsample_flag of each level.

Accordingly, by the leva box of FIG. 37, the level i each subrepresentation element has can be associated with the ith subsample fromthe top in which information is described in the subs box whose flags is0. Further, by the description of the subs box, the ith subsample fromthe top in which information is described in the subs box can bespecified.

By describing 5 as assignment_type of the leva box in such a manner asdescribed above, a level and a subsample described in the MPD file canbe associated with each other. Accordingly, 5 as assignment_type can besaid as information that associates a level and a subsample with eachother.

It is to be noted that two subsamples disposed on the track (quality1track) may be grouped into one group. In this case, a subsample groupentry (Sub-SampleGroupEntry) depicted in FIG. 38 is disposed in the trakbox of the track (quality1 track).

The subsample group entry of FIG. 38 is an expanded one ofSampleGroupDescriptionEntry on which not only a subsample in which animage is to be stored but a subsample group entry of a subsample inwhich any other than an image is to be stored are based.

In the subsample group entry of FIG. 38, a type (in the example of FIG.38, sgss) of the group of the subsample is described. Further, in thesubsample group entry of FIG. 38, code_parameter that is a name of asample entry of a subsample belonging to the group and sub_sample_flagsthat is flags of the subs box are described.

Further, as depicted in FIG. 39, assignment_type of the leva box is setto 0 that represents that the type of a thing associated with the levelis information relating to a sample belonging to a predetermined groupdescribed in an sgpd box (sample group description box).

Furthermore, in the leva box, not subsample_flag but grouping_typedescribed in the sgpd box associated with the level i (i=1, 2) isdescribed. grouping_type is a type of a group of a sub subsamplecorresponding to the sgpd box. In the example of FIG. 36, the type of agroup to which two subsamples disposed on the track (quality1 track)belong is sgss, and grouping_type corresponding to each level of theleva box is sgss for both of them.

By the leva box of FIG. 39, the level i each sub representation elementhas is associated with information described in the sgpd box andrelating to the ith subsample from the top in the information relatingto a subsample belonging to a group whose grouping_type is sgss.

As the information relating to a subsample, the name of the sample entryof the sample configuring the subsample and flags of the subs box aredescribed. In the example of FIG. 36, both of the names of the sampleentries of the sample configuring the two subsamples disposed on thetrack (quality1 track) are vqme, and flags of both of them is zero.Accordingly, in the sgpd box, vqme and 0 are described as theinformation relating to each subsample.

By the description of the sgpd box, the level i each sub representationelement has can be associated with the ith subsample from the top whoseflag is 0 and in which information is described in the subs box of thesample in which the name of the sample entry is vqme. Then, by thedescription of the subs box, the ith subsample from the top in whichinformation is described in the subs box can be specified.

In this manner, a level and a subsample described in an MPD file can beassociated with each other by the leva box of FIG. 39.

It is to be noted that, in the eighth embodiment, a quality file may bedivided as in the second to sixth embodiments.

Further, although, in the first to eighth embodiments, a subrepresentation element is expanded such that associationID can bedescribed, the sub representation element may be expanded such that anattribute other than associationId can be described.

For example, an attribute such as SubassociationId or the like in whichRepresentationID corresponding to a sub representation element isdescribed may be defined newly and the sub representation element may beexpanded such that such attribute can be described.

For example, the sub representation element may be expanded such thatdependencyId that is attribute indicative of an ID of a track thatrequires reference upon decoding can be described also in the subrepresentation element, and RepresentationID corresponding to the subrepresentation element may be described as dependencyId of the subrepresentation element.

It is to be noted that attribute of the sub representation elementindicative of a correspondence relationship between the subrepresentation element and RepresentationID can be used also when arelationship between a texture image and a depth occlusion image or arelationship other than the relationship between a depth occlusion imageand quality information is to be represented.

Ninth Embodiment

(Description of Computer to which Present Disclosure is Applied)

While the series of processes described above can be executed byhardware, it may otherwise be executed by software. Where the series ofprocesses is executed by software, a program that constructs thesoftware is installed into a computer. Here, the computer includes acomputer incorporated in hardware for exclusive use, for example, apersonal computer for universal use that can execute various functionsby installing various programs, and so forth.

FIG. 40 is a block diagram depicting an example of a configuration ofhardware of a computer that executes the series of processes describedhereinabove in accordance with a program.

In a computer 200, a CPU (Central Processing Unit) 201, a ROM (Read OnlyMemory) 202, and a RAM (Random Access Memory) 203 are connected to eachother by a bus 204.

To the bus 204, an input/output interface 205 is connected further. Tothe input/output interface 205, an inputting unit 206, an outputtingunit 207, a storage unit 208, a communication unit 209, and a drive 210are connected.

The inputting unit 206 is configured from a keyboard, a mouse, amicrophone and so forth. The outputting unit 207 is configured from adisplay, a speaker and so forth. The storage unit 208 is configured froma hard disk, a nonvolatile memory and so forth. The communication unit209 is configured from a network interface and so forth. The drive 210drives a removable medium 211 such as a magnetic disk, an optical disk,a magneto-optical disk, a semiconductor memory or the like.

In the computer 200 configured in such a manner as described above, theCPU 201 loads a program stored, for example, in the storage unit 208into the RAM 203 through the input/output interface 205 and the bus 204and executes the program to perform the series of processes describedhereinabove.

The program executed by the computer 200 (CPU 201) can be recorded onand provided as the removable medium 211, for example, as a packagemedium or the like. Further, the program can be provided through a wiredor wireless transmission medium such as a local area network, theInternet, or a digital satellite broadcast.

In the computer 200, the program can be installed into the storage unit208 through the input/output interface 205 by loading the removablemedium 211 into the drive 210. Further, the program can be received bythe communication unit 209 through a wired or wireless transmissionmedium and installed into the storage unit 208. Furthermore, it ispossible to install the program in advance into the ROM 202 or thestorage unit 208.

It is to be noted that the program executed by the computer 200 may be aprogram by which processes are performed in a time series in accordancewith an order described in the present specification or a program inwhich processes are performed in parallel or are performed at a timingat which the program is called or the like.

It is to be noted that, in the present specification, the term systemsignifies a set of a plurality of components (apparatus, modules (parts)and so forth) and it does not matter whether or not all components areprovided in the same housing. Accordingly, either of a plurality ofapparatuses accommodated in separate housings and connected to eachother through a network and one apparatus in which a plurality ofmodules are accommodated in one housing configures a system.

Further, the effects described in the present specification areexemplary only and shall not be restrictive, and other effects may beavailable.

Furthermore, the embodiment of the present disclosure is not limited tothe embodiments described hereinabove and can be altered in variousmanners without departing from the subject matter of the presentdisclosure.

For example, the segment file of occlusion images may not be generated.In particular, the present disclosure can be applied to an informationprocessing system in which a segment file including only depth images ordepth-related images including both depth images and occlusion images,namely, depth-related images including at least depth images, isgenerated.

It is to be noted that the present disclosure can have suchconfigurations as described below

(1)

A file generation apparatus including:

a file generation unit configured to generate a file in which qualityinformation representative of quality of a depth-related image includingat least a depth image is disposed in a form divided for each kind.

(2)

The file generation apparatus according to (1) above, in which

the file generation apparatus is configured such that

the file generation unit divides the quality information for each kindand disposes the divisional quality information into different files.

(3)

The file generation apparatus according to (2) above, in which

the file generation apparatus is configured such that

the depth-related image includes an occlusion image that is a textureimage of an occlusion region corresponding to the depth image.

(4)

The file generation apparatus according to (3) above, in which

the file generation apparatus is configured such that

the quality information of the occlusion image is informationrepresentative of a ratio of the occlusion region to a screen of thetexture image or a noise amount of the occlusion image.

(5)

The file generation apparatus according to (3) or (4) above, in which

the file generation apparatus is configured such that

the kind is a kind of the depth-related image.

(6)

The file generation apparatus according to any one of (2) to (4) above,in which

the file generation apparatus is configured such that

the kind is a texture image corresponding to the depth-related image.

(7)

The file generation apparatus according to (2) or (3) above, in which

the file generation apparatus is configured such that

the quality information is information representative of quality of a 3Dimage reproduced using a texture image corresponding to thedepth-related image and the depth-related image.

(8)

The file generation apparatus according to any one of (2) to (7) above,in which

the file generation apparatus is configured such that,

in a case where the file generation unit divides the quality informationof the depth-related images of a plurality of bit rates for each kindand disposes the divisional quality information into the differentfiles, the file generation unit disposes a plurality of kinds of thequality information disposed in a same file into tracks different fromeach other.

(9)

The file generation apparatus according to any one of (2) to (5) above,in which

the file generation apparatus is configured such that

in a case where the file generation unit divides the quality informationof the depth-related images of a plurality of bit rates for each kindand disposes the divisional quality information into the differentfiles, the file generation unit collectively samples a plurality ofpieces of the quality information disposed in a same file.

(10)

The file generation apparatus according to (9) above, in which

the file generation apparatus is configured such that

the plurality of pieces of quality information are individually dividedinto subsamples that are different from each other and are disposed intothe sample, and

the file generation unit describes a corresponding relationship betweenthe subsamples and track specification information for specifying atrack of the depth-related image corresponding to the qualityinformation disposed in the subsamples.

(11)

The file generation apparatus according to (10) above, in which

the file generation apparatus is configured such that

the file generation unit describes the correspondence relationship intothe file in order of disposing in the sample by describing trackspecification information of the depth-related image corresponding tothe subsamples into the file.

(12)

The file generation apparatus according to (10) or (11) above, in which

a track of the depth-related image corresponding to the qualityinformation disposed in the subsamples is disposed in the file same asthe file in which a track of the quality information is disposed.

(13)

A file generation method, in which

a file generation apparatus

includes a file generation step of generating a file in which qualityinformation representative of quality of a depth-related image includingat least a depth image is disposed in a form divided for each kind.

(14)

A reproduction apparatus including:

an acquisition unit configured to acquire, from a file in which qualityinformation representative of quality of a depth-related image includingat least a depth image is disposed in a form divided for each kind, thequality information of a given kind.

(15)

A reproduction method, in which

a reproduction apparatus

includes a an acquisition step of acquiring, from a file in whichquality information representative of quality of a depth-related imageincluding at least a depth image is disposed in a form divided for eachkind, the quality information of a given kind.

REFERENCE SIGNS LIST

11 File generation apparatus, 14 Video reproduction terminal, 83 Segmentfile generation unit, 84 MPD file generation unit, 103 Qualityinformation acquisition unit, 105 Image acquisition unit, 106 Decodingunit, 107 Output controlling unit

1. A file generation apparatus comprising: a file generation unit configured to generate a file in which quality information representative of quality of a depth-related image including at least a depth image is disposed in a form divided for each kind.
 2. The file generation apparatus according to claim 1, wherein the file generation apparatus is configured such that the file generation unit divides the quality information for each kind and disposes the divisional quality information into different files.
 3. The file generation apparatus according to claim 2, wherein the file generation apparatus is configured such that the depth-related image includes an occlusion image that is a texture image of an occlusion region corresponding to the depth image.
 4. The file generation apparatus according to claim 3, wherein the file generation apparatus is configured such that the quality information of the occlusion image is information representative of a ratio of the occlusion region to a screen of the texture image or a noise amount of the occlusion image.
 5. The file generation apparatus according to claim 3, wherein the file generation apparatus is configured such that the kind is a kind of the depth-related image.
 6. The file generation apparatus according to claim 2, wherein the file generation apparatus is configured such that the kind is a texture image corresponding to the depth-related image.
 7. The file generation apparatus according to claim 2, wherein the file generation apparatus is configured such that the quality information is information representative of quality of a 3D image reproduced using a texture image corresponding to the depth-related image and the depth-related image.
 8. The file generation apparatus according to claim 2, wherein the file generation apparatus is configured such that, in a case where the file generation unit divides the quality information of the depth-related images of a plurality of bit rates for each kind and disposes the divisional quality information into the different files, the file generation unit disposes a plurality of kinds of the quality information disposed in a same file into tracks different from each other.
 9. The file generation apparatus according to claim 2, wherein the file generation apparatus is configured such that in a case where the file generation unit divides the quality information of the depth-related images of a plurality of bit rates for each kind and disposes the divisional quality information into the different files, the file generation unit collectively samples a plurality of kinds of the quality information disposed in a same file and disposes the plurality of kinds of the quality information into a same track.
 10. The file generation apparatus according to claim 9, wherein the file generation apparatus is configured such that the plurality of kinds of quality information are individually divided into subsamples that are different from each other and are disposed into the sample, and the file generation unit describes, into the file, a corresponding relationship between the subsamples and track specification information for specifying a track of the depth-related image corresponding to the quality information disposed in the subsamples.
 11. The file generation apparatus according to claim 10, wherein the file generation apparatus is configured such that the file generation unit describes the correspondence relationship into the file by describing track specification information of the depth-related image corresponding to the subsamples in an order in which the subsamples are disposed in the sample into the file.
 12. The file generation apparatus according to claim 10, wherein a track of the depth-related image corresponding to the quality information disposed in the subsamples is disposed in the file same as the file in which a track of the quality information is disposed.
 13. A file generation method comprising: a file generation step of a file generation apparatus generating a file in which quality information representative of quality of a depth-related image including at least a depth image is disposed in a form divided for each kind.
 14. A reproduction apparatus comprising: an acquisition unit configured to acquire, from a file in which quality information representative of quality of a depth-related image including at least a depth image is disposed in a form divided for each kind, the quality information of a given kind.
 15. A reproduction method comprising: an acquisition step of a reproduction apparatus acquiring, from a file in which quality information representative of quality of a depth-related image including at least a depth image is disposed in a form divided for each kind, the quality information of a given kind. 